Bug in Remoting 2.0

[hceylan]

I think I have spotted a bug in Remoting 2.0.

I use remoting 2.0 coupled with Jboss 4.0.4 hosted EJB3 container.

After the upgrade to remoting 2.0, I started getting calls for clients stopping to repond users.

Exception was thrown from: Microcontainer Line 706.


throw new SocketException("Can not obtain client socket connection from pool. Have waited " +
numberOfRetries + " seconds for available connection (" + usedPooled + " in use)");


Having investigated the code I have found that if a client waits for some time so long as server times out the connection and closes the socket, remoting client gets the very same socket from the pool, incresing usedPooled by one.

Then in the transport writes the invocation to the socket and checks back the response. However due to the socket being closed by the server, socketexception is received and number of tries is incremented by starting over by getting a socket from the pool.

This simply means that socket sis dropped without decrementing the use count kept in "usedPooled "

Therefore, after some time usedPooled overflows the max which is by default 50 and client process cannot do remoting any more.

Just before numberof tries exhausts in the transport method, flushConnectionPool is called. IMHO this has no point at all. it mereley disposes the pooled sockets. But every numberOfCallRetries times timed out socket is leaked.

My recommendation is to insert

usedPooled--;

into exception handling at line 359.

Upon confirmation I may open up a JIRA and post the formal patch.

Hasan Ceylan

[ron_sigal]

Hi Hasan,

"hasan" wrote:

I think I have spotted a bug in Remoting 2.0.

I believe you're right.

"hasan" wrote:

Having investigated the code I have found that if a client waits for some time so long as server times out the connection and closes the socket, remoting client gets the very same socket from the pool, incresing usedPooled by one.

I'm not sure how this scenario can occur. If the parameter "socket.check_connection" is configured to "false", then the server shouldn't time out a socket (unless it's in the middle of processing an invocation). If it is configured to "true", then getPooledConnection() should see that the connection isn't functional and shouldn't return the socket. Is it possible that you have "socket.check_connection=true" on the server and "socket.check_connection=false" on the client? This inconsistency could occur if you configure the Connector in a service.xml file:

 <mbean code="org.jboss.remoting.transport.Connector"
 name="jboss.messaging:service=Connector,transport=socket"
 display-name="Socket transport Connector">
 <attribute name="Configuration">
 <config>
 <invoker transport="socket">
 ...
 <attribute name="socket.check_connection" isParam="true">false</attribute>
 ...
 </invoker>>
 </config>
 </attribute>
 </mbean>

but leave out "isParam=\"true\"". Could you check that?

"hasan" wrote:

This simply means that socket sis dropped without decrementing the use count kept in "usedPooled "

Therefore, after some time usedPooled overflows the max which is by default 50 and client process cannot do remoting any more.
...

My recommendation is to insert

usedPooled--;

into exception handling at line 359.

Indeed, lines 362 and 385 both have the effect of eliminating a socket without decrementing the count.

Hasan, if you do the honors of posting a JIRA issue, I will see that the correction is made.

[hceylan]

Hello Ron,

Thanks for the reply. I will test the param solution then ost my reply and create a JIRA for this issue.

Regards,
Hasan Ceylan

[ron_sigal]

I have created a JIRA issue, http://jira.jboss.com/jira/browse/JBREM-625, and made the necessary corrections.

Thank you, Hasan!

[hceylan]

Hi Ron,

I have quite busy since then. It is a shame I haven't been able to test your proposal. Sorry,

Nevertheless, thanks for the JIRA and solution.

Hasan