Remoting 2: Timeout waiting on socket for InvokerLocator

[alrubinger]

Hi guys, I have a JMS/Remoting error that's environment-specific to my Hudson, cannot reproduce locally or on other CI servers. Mostly likely I just need to make the test more resilient?

java.util.concurrent.ExecutionException: org.jboss.jms.exception.MessagingNetworkFailureException
 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:205)
 at java.util.concurrent.FutureTask.get(FutureTask.java:80)
 at org.jboss.ejb3.test.strictpool.unit.StrictPoolUnitTestCase.testMessageDriven(StrictPoolUnitTestCase.java:123)
 at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
 at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
 at junit.extensions.TestSetup.run(TestSetup.java:27)
Caused by: org.jboss.jms.exception.MessagingNetworkFailureException
 at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:240)
 at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205)
 at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:495)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeTarget(ClientSessionDelegate$send_6145266547759487588.java)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
 at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661)
 at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleSend_20120943.invoke(SessionAspect_z_handleSend_20120943.java)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
 at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
 at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:269)
 at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect_z_handleSend_20120943.invoke(ProducerAspect_z_handleSend_20120943.java)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
 at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
 at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
 at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
 at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
 at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
 at org.jboss.ejb3.test.strictpool.unit.MDBCallable.call(MDBCallable.java:79)
 at org.jboss.ejb3.test.strictpool.unit.MDBCallable.call(MDBCallable.java:45)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
 at java.util.concurrent.FutureTask.run(FutureTask.java:123)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 at java.lang.Thread.run(Thread.java:595)
Caused by: org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://localhost:4457/?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&connectionWait=10&dataType=jms&marshaller=org.jboss.jms.wireformat.JMSWireFormat&numberOfCallRetries=1&pingFrequency=214748364&pingWindowFactor=10&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&validatorPingPeriod=10000&validatorPingTimeout=5000]
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:771)
 at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:423)
 at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:160)
 at org.jboss.remoting.Client.invoke(Client.java:1708)
 at org.jboss.remoting.Client.invoke(Client.java:612)
 at org.jboss.remoting.Client.invokeOneway(Client.java:660)
 at org.jboss.remoting.Client.invokeOneway(Client.java:848)
 at org.jboss.remoting.Client.invokeOneway(Client.java:838)
 at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180)
 ... 32 more
Caused by: java.lang.IllegalStateException: Timeout waiting for a free socket
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1048)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:757)
 ... 40 more

Any recommendations for how I might increase the timeout? Client code is pretty straightforward:

QueueSession session = connection.createQueueSession(false, QueueSession.AUTO_ACKNOWLEDGE);
 try
 {
 TemporaryQueue replyQueue = session.createTemporaryQueue();
 QueueReceiver receiver = session.createReceiver(replyQueue);

 QueueSender sender = session.createSender(queue);
 sender.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
 TextMessage message = session.createTextMessage();
 message.setJMSReplyTo(replyQueue);
 message.setText(text);
 sender.send(message); < Exception here, MDBCallable.java:79
 sender.close(); ....

Thanks as always.

S,
ALR

[ron_sigal]

That particular timeout value is specified by the "connectionWait" parameter, which defaults to 30000, so you could add, for example,

<attribute name="connectionWait" isParam="true">60000</attribute>

to remoting-bisocket-service.xml, JBM's configuration file.

Note that for this timeout to occur, the MicroSocketClientInvoker must already have 50 (by default) connections already in use, before waiting 30 seconds for one of them to return to the connection pool. Does that sound plausible for your test?

[wolfc]

It would be nice to have the debug log show traces of all the creation points of current connections in the pool. This should really be a configuration option and a debug level 'trace', because storing stack trace for fun is expensive.

[sentcs]

Hi,

I am also getting same problem for jboss-4.2.3.GA-jdk6 with jboss-messaging-1.4.0.SP3 envirnment.

As per your suggestion, I have added one more attribute as 'connectionwait' into remoting-bisocket-service.xml.

Here i have a query as 'how to increase the number of connections' since you said that connection default is 50.

And i want know, any impact if i am going to change the number of connection.

Thanks.

[ron_sigal]

"sentcs" wrote:

how to increase the number of connections

Normally, the default value is 50, as I said, and, normally, the parameter to set is "clientMaxPoolSize". However, setting that value in JBossMessaging's remoting-bisocket-service.xml would also affect callback clients on the server side, which the JBM guys didn't want to do, so they created a JBM specific parameter, "JBM_clientMaxPoolSize". Also, I see that they set it to "200" in remoting-bisocket-service.xml.

"sentcs" wrote:

And i want know, any impact if i am going to change the number of connection

If you are actually exhausting the pool, and you increase the pool size, then you could get more activity on the client, for better or worse. Also, a single client could take up more of the server's resources. If the server isn't overloaded, then you might consider increasing the number of worker threads on the server, using the "maxPoolSize" parameter. The default is 300.