4 Replies Latest reply on Feb 4, 2009 11:52 PM by Ron Sigal

    Remoting 2: Timeout waiting on socket for InvokerLocator

    Andrew Rubinger Master

      Hi guys, I have a JMS/Remoting error that's environment-specific to my Hudson, cannot reproduce locally or on other CI servers. Mostly likely I just need to make the test more resilient?

      java.util.concurrent.ExecutionException: org.jboss.jms.exception.MessagingNetworkFailureException
       at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:205)
       at java.util.concurrent.FutureTask.get(FutureTask.java:80)
       at org.jboss.ejb3.test.strictpool.unit.StrictPoolUnitTestCase.testMessageDriven(StrictPoolUnitTestCase.java:123)
       at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
       at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
       at junit.extensions.TestSetup.run(TestSetup.java:27)
      Caused by: org.jboss.jms.exception.MessagingNetworkFailureException
       at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:240)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:495)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeTarget(ClientSessionDelegate$send_6145266547759487588.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
       at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661)
       at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleSend_20120943.invoke(SessionAspect_z_handleSend_20120943.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
       at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:269)
       at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect_z_handleSend_20120943.invoke(ProducerAspect_z_handleSend_20120943.java)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
       at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
       at org.jboss.ejb3.test.strictpool.unit.MDBCallable.call(MDBCallable.java:79)
       at org.jboss.ejb3.test.strictpool.unit.MDBCallable.call(MDBCallable.java:45)
       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
       at java.util.concurrent.FutureTask.run(FutureTask.java:123)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      Caused by: org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://localhost:4457/?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&connectionWait=10&dataType=jms&marshaller=org.jboss.jms.wireformat.JMSWireFormat&numberOfCallRetries=1&pingFrequency=214748364&pingWindowFactor=10&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&validatorPingPeriod=10000&validatorPingTimeout=5000]
       at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:771)
       at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:423)
       at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:160)
       at org.jboss.remoting.Client.invoke(Client.java:1708)
       at org.jboss.remoting.Client.invoke(Client.java:612)
       at org.jboss.remoting.Client.invokeOneway(Client.java:660)
       at org.jboss.remoting.Client.invokeOneway(Client.java:848)
       at org.jboss.remoting.Client.invokeOneway(Client.java:838)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180)
       ... 32 more
      Caused by: java.lang.IllegalStateException: Timeout waiting for a free socket
       at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1048)
       at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:757)
       ... 40 more


      Any recommendations for how I might increase the timeout? Client code is pretty straightforward:

      QueueSession session = connection.createQueueSession(false, QueueSession.AUTO_ACKNOWLEDGE);
       try
       {
       TemporaryQueue replyQueue = session.createTemporaryQueue();
       QueueReceiver receiver = session.createReceiver(replyQueue);
      
       QueueSender sender = session.createSender(queue);
       sender.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
       TextMessage message = session.createTextMessage();
       message.setJMSReplyTo(replyQueue);
       message.setText(text);
       sender.send(message); < Exception here, MDBCallable.java:79
       sender.close(); ....


      Thanks as always.

      S,
      ALR

        • 1. Re: Remoting 2: Timeout waiting on socket for InvokerLocator
          Ron Sigal Master

          That particular timeout value is specified by the "connectionWait" parameter, which defaults to 30000, so you could add, for example,

          <attribute name="connectionWait" isParam="true">60000</attribute>
          


          to remoting-bisocket-service.xml, JBM's configuration file.

          Note that for this timeout to occur, the MicroSocketClientInvoker must already have 50 (by default) connections already in use, before waiting 30 seconds for one of them to return to the connection pool. Does that sound plausible for your test?

          • 2. Re: Remoting 2: Timeout waiting on socket for InvokerLocator
            Carlo de Wolf Master

            It would be nice to have the debug log show traces of all the creation points of current connections in the pool. This should really be a configuration option and a debug level 'trace', because storing stack trace for fun is expensive.

            • 3. Re: Remoting 2: Timeout waiting on socket for InvokerLocator
              senthil kumar Newbie

              Hi,

              I am also getting same problem for jboss-4.2.3.GA-jdk6 with jboss-messaging-1.4.0.SP3 envirnment.

              As per your suggestion, I have added one more attribute as 'connectionwait' into remoting-bisocket-service.xml.

              Here i have a query as 'how to increase the number of connections' since you said that connection default is 50.

              And i want know, any impact if i am going to change the number of connection.

              Thanks.

              • 4. Re: Remoting 2: Timeout waiting on socket for InvokerLocator
                Ron Sigal Master

                 

                "sentcs" wrote:

                how to increase the number of connections


                Normally, the default value is 50, as I said, and, normally, the parameter to set is "clientMaxPoolSize". However, setting that value in JBossMessaging's remoting-bisocket-service.xml would also affect callback clients on the server side, which the JBM guys didn't want to do, so they created a JBM specific parameter, "JBM_clientMaxPoolSize". Also, I see that they set it to "200" in remoting-bisocket-service.xml.

                "sentcs" wrote:

                And i want know, any impact if i am going to change the number of connection


                If you are actually exhausting the pool, and you increase the pool size, then you could get more activity on the client, for better or worse. Also, a single client could take up more of the server's resources. If the server isn't overloaded, then you might consider increasing the number of worker threads on the server, using the "maxPoolSize" parameter. The default is 300.