-
1. Re: Problem with backlog queue behavior ?
nicolas.medoc Jan 18, 2008 6:02 AM (in response to nicolas.medoc)Finally I have downloaded the 4.2.2 version of JBoss and the 2.2.2.SP3 version of Remoting.
I have proceed with the same approach as previously, using the folowing configuration :<invoker transport="socket"> <attribute name="serverBindAddress">${jboss.bind.address}</attribute> <attribute name="serverBindPort">3873</attribute> <attribute name="maxPoolSize">400</attribute> <attribute name="backlog">200</attribute> <attribute name="clientMaxPoolSize" isParam="true">404</attribute> <attribute name="timeout" isParam="true">50000</attribute> </invoker>
With this new version, the clientMaxPoolSize must be increase up to maxPoolSize for testing the backlog queue (there was not the case with 1.4.3 version of remoting, it was a bug ?).
When 500 threads are created on the client, the following behavior is observed : around the first 400 client threads, all finish and works fine. Then in between around the 400th and the 500th client thread, 3 of these (sometime 1, sometime 2) throws an exception :org.jboss.remoting.CannotConnectException:Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://127.0.0.1:3873/?clientMaxPoolSize=404&timeout=50000] caused by : java.lang.reflect.InvocationTargetException:null caused by : java.net.SocketTimeoutException:Read timed out
A server side exception is thrown :ERROR [ServerThread] Worker thread initialization failure java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:168) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:235) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200) at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2380) at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2447) at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2519) at java.io.ObjectInputStream.read(ObjectInputStream.java:789) at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
when the clientMaxPoolSize is increased up to 450, there are around 50 client threads that failes.
On the JMX consol the currentClientPoolSize value is 400 until the socket timeout delay has passed. Then the 400 server threads return on the pool.
The maxPoolSize + backlog is sufficient to accept the 500 thread connections. So I don't understand why method invokations failes when they waits in the backlog queue (or in the "accept thread"). When the client has finish, the server thread should release the connection and return in the pool. (but it isn't...why?)
What is the problem in my configuration ?
What is the good approach for managing "server side" bursts client connections ?
PS: When the number of concurrent client threads (and also clientMaxPoolThread) is greater than maxPoolSize + backlog (ie 550), the same SocketTimeoutException is thrown. However when it is lesser, there are no fails.
Thanks for your help.
Nicolas Medoc. -
2. Re: Problem with backlog queue behavior ?
ron_sigal Feb 7, 2008 5:11 AM (in response to nicolas.medoc)Hi Nicolas,
Couple of points:
1. clientMaxPoolSize=50 means that there can never be more than 50 simultaneous invocations from a given client.
2. maxPoolSize=400 means that there can never be more than 400 simultaneous invocations on the server.
3. 1) and 2) imply that, in your first example, you never came close to using the ServerSocket queue (backlog).
4.
When the client has finish, the server thread should release the connection and return in the pool.
isn't quite true. When a ServerThread finishes an invocation and returns a result, the client thread will return to the pool, but it maintains a connection to the ServerThread. That is, the ServerThread can only be used by the client thread that originally connected to it, at least until a socket timeout occurs. What is happening is that the 401st invocation sees that all the client threads are busy but, since clientMaxPoolSize=404, it tries to create a new connection. However, on the server side, all of the ServerThreads are still in use. The AcceptorThread will try to "evict" one of the ServerThreads and try to reuse it, but, frankly, that's old code and I don't believe it works. In fact, I have tried to fix the eviction process in Remoting 2.4.0.Beta1. So, I believe, the AcceptorThread will hang until a ServerThread experiences a socket timeout and returns itself to the pool. It looks like the 401st client socket has timed out by that time, and then when the ServerThread finally tries to talk to the 401st socket, it finds the "connection reset".
So, setting clientMaxPoolSize > maxPoolSize is not a good idea, at least for Remoting 2.2.x. However, if you set clientMaxPoolSize=maxPoolSize=400 and create 500 simultaneous invocations with a duration less than the configured timeout value, I would expect all the invocations to work. Can you verify that? -
3. Re: Problem with backlog queue behavior ?
nicolas.medoc Feb 22, 2008 7:56 AM (in response to nicolas.medoc)Hi Ron,
Thanks for your response.
When the clientMaxPoolSize=MaxPoolSize=100, all the 300 client threads works fine - only for the threads that don't reach numberOfRetries=30 (corresponding to a time of 30 seconds).
Ok, there is sense to have clientMaxPoolSize<=maxpoolSize. But in this case I should expect that the backlog queue don't be used. So I wonder about the responsability of backlog queue. In what situation is it used ?
PS : concerning my tests whith the 1.4.3 version of Remoting, I believe that the clientMaxPoolSize didn't works (contrary to the 2.2.2 version). I had observed that the first 400 serverThreads had been created simultanously although the clientMaxPoolSize=50. It is not a problem for me because I'm using the 2.2.2 version. -
4. Re: Problem with backlog queue behavior ?
ron_sigal Apr 23, 2008 2:16 AM (in response to nicolas.medoc)"nicolas.medoc" wrote:
Ok, there is sense to have clientMaxPoolSize<=maxpoolSize. But in this case I should expect that the backlog queue don't be used. So I wonder about the responsability of backlog queue. In what situation is it used ?
Well, SocketServerInvoker will stop accepting connections when all the ServerThreads are in use, so if you have clientMaxPoolSize > maxpoolSize and create too many connection requests, the requests will go on the backlog queue. It just isn't a good situation, though.