Problem with backlog queue behavior ?

[nicolas.medoc]

hello

My JBoss version is 4.0.4GA. The JBoss Remoting version is 1.4.3 (I can see it in boss-remoting.jar : meta-inf/manifest.mf).For testing my configuration, I create 500 threads in my client. Each thread invoke an EJB method that simulate an execution with Thread.sleep(time).

My configuration is the follow :

 <invoker transport="socket">
 <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
 <attribute name="serverBindPort">3873</attribute>
 <attribute name="maxPoolSize">400</attribute>
 <attribute name="backlog">200</attribute>
 <attribute name="clientMaxPoolSize" isParam="true">50</attribute>
 <attribute name="timeout" isParam="true">60000</attribute>
 <attribute name="numberOfRetries" isParam="true">60</attribute>
 </invoker>

I have volontary set the maxpoolSize to 400 for testing the server side behavior of waiting queue (parameterized with 'backlog' value).

When the sleep time is greater than 3000 ms, one or more client threads throws an exception :

java.lang.reflect.UndeclaredThrowableException:null

caused by : java.rmi.MarshalException:Socket timed out. Waited 60000 milliseconds for response while calling on InvokerLocator [socket://10.20.1.113:3873/?clientMaxPoolSize=50&numberOfRetries=60&timeout=60000]; nested exception is: java.net.SocketTimeoutException: Read timed out

caused by : java.net.SocketTimeoutException:Read timed out

When all client threads have finished, I see that on the JMX Console the value of currentClientPoolSize > 0. Its value is exactly the same that number of client threads that failed with SocketTimeoutException.

I conclued that the first 400 server threads work fine (corresponding to the maxPoolSize). Then, some of the next 100 server threads never return in an "available" state.

Is it a bug on the 1.4.3 version or a configuration problem ?

Thanks for your help.

Nicolas Medoc.

[nicolas.medoc]

Finally I have downloaded the 4.2.2 version of JBoss and the 2.2.2.SP3 version of Remoting.
I have proceed with the same approach as previously, using the folowing configuration :

 <invoker transport="socket">
 <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
 <attribute name="serverBindPort">3873</attribute>
 <attribute name="maxPoolSize">400</attribute>
 <attribute name="backlog">200</attribute>
 <attribute name="clientMaxPoolSize" isParam="true">404</attribute>
 <attribute name="timeout" isParam="true">50000</attribute>
 </invoker>

With this new version, the clientMaxPoolSize must be increase up to maxPoolSize for testing the backlog queue (there was not the case with 1.4.3 version of remoting, it was a bug ?).
When 500 threads are created on the client, the following behavior is observed : around the first 400 client threads, all finish and works fine. Then in between around the 400th and the 500th client thread, 3 of these (sometime 1, sometime 2) throws an exception :

 org.jboss.remoting.CannotConnectException:Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://127.0.0.1:3873/?clientMaxPoolSize=404&timeout=50000]
 caused by : java.lang.reflect.InvocationTargetException:null
 caused by : java.net.SocketTimeoutException:Read timed out

A server side exception is thrown :

ERROR [ServerThread] Worker thread initialization failure
java.net.SocketException: Connection reset
 at java.net.SocketInputStream.read(SocketInputStream.java:168)
 at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
 at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
 at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200)
 at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2380)
 at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2447)
 at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2519)
 at java.io.ObjectInputStream.read(ObjectInputStream.java:789)
 at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

when the clientMaxPoolSize is increased up to 450, there are around 50 client threads that failes.
On the JMX consol the currentClientPoolSize value is 400 until the socket timeout delay has passed. Then the 400 server threads return on the pool.

The maxPoolSize + backlog is sufficient to accept the 500 thread connections. So I don't understand why method invokations failes when they waits in the backlog queue (or in the "accept thread"). When the client has finish, the server thread should release the connection and return in the pool. (but it isn't...why?)

What is the problem in my configuration ?
What is the good approach for managing "server side" bursts client connections ?

PS: When the number of concurrent client threads (and also clientMaxPoolThread) is greater than maxPoolSize + backlog (ie 550), the same SocketTimeoutException is thrown. However when it is lesser, there are no fails.


Thanks for your help.
Nicolas Medoc.

[ron_sigal]

Hi Nicolas,

Couple of points:

1. clientMaxPoolSize=50 means that there can never be more than 50 simultaneous invocations from a given client.

2. maxPoolSize=400 means that there can never be more than 400 simultaneous invocations on the server.

3. 1) and 2) imply that, in your first example, you never came close to using the ServerSocket queue (backlog).

4.

When the client has finish, the server thread should release the connection and return in the pool.

isn't quite true. When a ServerThread finishes an invocation and returns a result, the client thread will return to the pool, but it maintains a connection to the ServerThread. That is, the ServerThread can only be used by the client thread that originally connected to it, at least until a socket timeout occurs. What is happening is that the 401st invocation sees that all the client threads are busy but, since clientMaxPoolSize=404, it tries to create a new connection. However, on the server side, all of the ServerThreads are still in use. The AcceptorThread will try to "evict" one of the ServerThreads and try to reuse it, but, frankly, that's old code and I don't believe it works. In fact, I have tried to fix the eviction process in Remoting 2.4.0.Beta1. So, I believe, the AcceptorThread will hang until a ServerThread experiences a socket timeout and returns itself to the pool. It looks like the 401st client socket has timed out by that time, and then when the ServerThread finally tries to talk to the 401st socket, it finds the "connection reset".

So, setting clientMaxPoolSize > maxPoolSize is not a good idea, at least for Remoting 2.2.x. However, if you set clientMaxPoolSize=maxPoolSize=400 and create 500 simultaneous invocations with a duration less than the configured timeout value, I would expect all the invocations to work. Can you verify that?

[nicolas.medoc]

Hi Ron,

Thanks for your response.

When the clientMaxPoolSize=MaxPoolSize=100, all the 300 client threads works fine - only for the threads that don't reach numberOfRetries=30 (corresponding to a time of 30 seconds).

Ok, there is sense to have clientMaxPoolSize<=maxpoolSize. But in this case I should expect that the backlog queue don't be used. So I wonder about the responsability of backlog queue. In what situation is it used ?


PS : concerning my tests whith the 1.4.3 version of Remoting, I believe that the clientMaxPoolSize didn't works (contrary to the 2.2.2 version). I had observed that the first 400 serverThreads had been created simultanously although the clientMaxPoolSize=50. It is not a problem for me because I'm using the 2.2.2 version.

[ron_sigal]

"nicolas.medoc" wrote:

Ok, there is sense to have clientMaxPoolSize<=maxpoolSize. But in this case I should expect that the backlog queue don't be used. So I wonder about the responsability of backlog queue. In what situation is it used ?

Well, SocketServerInvoker will stop accepting connections when all the ServerThreads are in use, so if you have clientMaxPoolSize > maxpoolSize and create too many connection requests, the requests will go on the backlog queue. It just isn't a good situation, though.