1 2 Previous Next 22 Replies Latest reply on Apr 21, 2010 6:32 AM by mingjun jiang

    Client hangs when get JMS connection factory

    Li Lin Newbie

      Sometimes, my client will hang when client try to get JMS connection factory from JBoss Messaging Server. Server side exception is thrown when issue occurred:

      2009-10-20 18:08:09,655 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure
      java.net.SocketException: Connection reset
       at java.net.SocketInputStream.read(SocketInputStream.java:168)
       at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
       at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
       at java.io.FilterInputStream.read(FilterInputStream.java:66)
       at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:859)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:545)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:406)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
      


      My JBoss Server Version is 4.2.3.GA, JBoss Remoting Version is 2.2.3 SP1, JBoss Messaging Version is 1.4.5 GA
      My configuration is
       <invoker transport="bisocket">
      
       <!-- There should be no reason to change these parameters - warning!
       Changing them may stop JBoss Messaging working correctly -->
       <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
       <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
       <attribute name="dataType" isParam="true">jms</attribute>
       <attribute name="socket.check_connection" isParam="true">false</attribute>
       <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
       <attribute name="serverBindPort">4457</attribute>
       <attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
       <attribute name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>
      
       <attribute name="numberOfCallRetries" isParam="true">1</attribute>
       <attribute name="pingFrequency" isParam="true">214748364</attribute>
       <attribute name="pingWindowFactor" isParam="true">10</attribute>
       <attribute name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>
       <!-- End immutable parameters -->
      
       <attribute name="stopLeaseOnFailure" isParam="true">true</attribute>
      
       <!-- Periodicity of client pings. Server window by default is twice this figure -->
       <attribute name="clientLeasePeriod" isParam="true">10000</attribute>
       <attribute name="validatorPingPeriod" isParam="true">10000</attribute>
       <attribute name="validatorPingTimeout" isParam="true">5000</attribute>
       <attribute name="registerCallbackListener">false</attribute>
      
       <attribute name="timeout" isParam="true">0</attribute>
      
       <!-- Number of seconds to wait for a connection in the client pool to become free -->
       <attribute name="numberOfRetries" isParam="true">10</attribute>
      
       <!-- Max Number of connections in client pool. This should be significantly higher than
       the max number of sessions/consumers you expect -->
       <attribute name="JBM_clientMaxPoolSize" isParam="true">200</attribute>
      
       <!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->
       <attribute name="callbackTimeout">10000</attribute>
       <attribute name="secondaryBindPort">4557</attribute>
      
       <!-- Use these parameters to specify values for binding and connecting control connections to
       work with your firewall/NAT configuration
       <attribute name="secondaryBindPort">xyz</attribute>
       <attribute name="secondaryConnectPort">abc</attribute>
       -->
      
       </invoker>
      


      After testing, I find,
      1) if client ethernet cable is pulled, CurrentClientPoolSize of invoker bisocket does not decrease.
      2) If kill client process, CurrentClientPoolSize decrease.
      3) If CurrentClientPoolSize becomes the same with value of MaxPoolSize, new request will go to backlog queue forever (parameter timeout is 0), so client hangs.


      I try to define parameter timeout as 10000, not 0, everything works fine even though client ethernet cable is pulled. It seems issue is resolved. But I got some information from JBoss web site.
      https://jira.jboss.org/jira/browse/JBMESSAGING-1268
      https://jira.jboss.org/jira/browse/JBREM-947
      https://jira.jboss.org/jira/browse/JBREM-949

      From information, I think daemon thread should use parameter validatorPingPeriod and validatorPingTimeout to remove dead thread, not timeout. But why will parameter timeout affect daemon thread? What's wrong with my understanding?


        • 1. Re: Client hangs when get JMS connection factory
          mingjun jiang Newbie

          Let me further elaborate this issue:

          In our application, there are some Message listeners running on the client side, these message listeners will receive messages from queue/topic deployed in JBoss Messaging

          Configuration:
          We created our own JMS Connection factory which uses the default remoting connector. As you know, the default remoting connector is configured to use the bisocket transport

          During we run our application, we open the JBoss web console to monitor the value of currentClientPoolSize under Jboss.remoting JMX MBean


          How to reproduce this issue:
          1. Run 5 message listeners in the client side to receive messages from JBoss Messaging, then we monitor the value of currentClientPoolSize, we observe the value is 10
          2. After processing several messages, we manually pull out the ethernet cable. After some time, we found the value of currentClientPoolSize is still 10, no any change. (If we killed the process of message listener in client side, then the value of currentClientPoolSize will decrease to 0 immediately)
          3. We run another 5 message listeners in client side, then the value of currentClientPoolSize will become 20
          4. After we do the same operations above several times, the value of currentClientPoolSize will increase continuously. Once the value of currentClientPoolSize is equal to the MaxPoolSize, then the subsequent incoming client requests will hang, and we will encounter the following exception in server side

          2009-10-20 18:08:09,655 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initi
          alization failure
          java.net.SocketException: Connection reset
           at java.net.SocketInputStream.read(SocketInputStream.java:168)
           at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
           at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
           at java.io.FilterInputStream.read(FilterInputStream.java:66)
           at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:859)
           at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:545)
           at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:406)
          at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
          


          Conclusion: JBoss Messaging will not close the failure connections if they are caused by manually pulling out ethernet cable. As a result, the value of currentClientPoolSize will increase continuously and finally the new client requests will hang

          We found several similar problems in JBoss JIRA as below:
          https://jira.jboss.org/jira/browse/JBMESSAGING-1268
          https://jira.jboss.org/jira/browse/JBREM-947
          https://jira.jboss.org/jira/browse/JBREM-949

          In the JBREM-947 and JBREM-949, the remoting wrongly depend on the socket timeout for failure detection, the bug reporter said 'Remoting should not be dependent on the socket timeout for failure detection, the connetion validation and socket timeout should be possible to be configured separately'. According to the JIRA, this issue had been fixed in Remoting 2.2.2. SP8 and 2.4.0.CR2 (Pinto), since it is said 'JBREM-947 - is the proper fix in remoting which will allow us to configure timeouts and validation interval separately'. (stated in JBMESSAGING-1268)

          However, according to our test, this issue was not fixed properly. We suspect the Remoting 2.2.3 still wrongly depend on the socket timeout for failure connection detection. So, we tried to change the timeout of bisocket transport from 0 (infinite) to a lower value e.g. 10 seconds, then we repeat the test steps again and observe the jboss web console, fortunately, we found the failure connection will be closed after 10 seconds. In JBREM-947, a daemon thread and validatorPingTimeout parameter are mentioned, we think daemon thread should use parameter validatorPingPeriod and validatorPingTimeout to close the failure connection instead of socket timeout. Thus, we post this to seek your expertise. Please correct us if we are wrong.



          • 2. Re: Client hangs when get JMS connection factory
            mingjun jiang Newbie

            Supplement:

            For the issue that connection won't be closed due to pulling out the ethernet cable, JBREM-949 said it was fixed in Remoting 2.2.2.SP8 , whereas JBREM-947 said it was fixed in 2.4.0.CR2 (Pinto). We are using Remoting 2.2.3 SP1, but we still encounter this issue. I wonder which version of remoting really fixed this issue. In my opinion, if it was fixed in Remoting 2.2.2 SP 8, then our Remoting 2.2.3 SP1 should work well.

            We also tried to use Remoting 2.4.0, but it's incompatible with Jboss Messaging 1.4.5 GA since we saw some exceptions were thrown when JBoss start up

            • 3. Re: Client hangs when get JMS connection factory
              Rakesh Naidu Newbie

              G'day guys

               

              I seem to be having similar issues too, so I hope the threadstarter doesn't mind me 'hitching a ride' on this thread.

              Firstly, my platform:

               

              OS:    Linux 2.6.5-7.308-smp (i386)

              JBOSS: 4.2.2.GA

              JBM:   1.4.4.GA

              JBREM: 2.2.3_SP1

               

              The moment I get this error in my SERVER.LOG, the NonDurableMessageCount starts increasing infinately:

               

              2009-12-15 18:14:56,181 ERROR [org.jboss.remoting.transport.socket.SocketClientInvoker] Got marshalling exception, exiting
              java.io.IOException: Connection reset
                   at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
                   at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
                   at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                   at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                   at java.io.DataOutputStream.flush(DataOutputStream.java:106)
                   at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:93)
                   at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)
                   at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:971)
                   at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:606)
                   at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:422)
                   at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
                   at org.jboss.remoting.Client.invoke(Client.java:1634)
                   at org.jboss.remoting.Client.invoke(Client.java:548)
                   at org.jboss.remoting.Client.invokeOneway(Client.java:598)
                   at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
                   at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
                   at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1454)
                   at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1366)
                   at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:327)
                   at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
                   at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:583)
                   at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
                   at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:638)
                   at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:506)
                   at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:239)
                   at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2212)
                   at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:499)
                   at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:763)
                   at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)
                   at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                   at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                   at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                   at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
                   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   at java.lang.reflect.Method.invoke(Method.java:597)
                   at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                   at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                   at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                   at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                   at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                   at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:866)
                   at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
                   at org.jboss.remoting.Client.invoke(Client.java:1634)
                   at org.jboss.remoting.Client.invoke(Client.java:548)
                   at org.jboss.remoting.Client.invoke(Client.java:536)
                   at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
                   at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
                   at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:499)
                   at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                   at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661)
                   at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect442.invoke(SessionAspect442.java)
                   at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                   at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                   at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                   at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                   at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                   at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
                   at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:269)
                   at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect484.invoke(ProducerAspect484.java)
                   at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                   at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                   at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                   at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
                   at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
                   at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
                   at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
                   at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
                   at genesis.cmms.bean.CMMSBean.publishEvent(CMMSBean.java:1116)
                   at genesis.cmms.bean.CMMSBean.processEMTEvent(CMMSBean.java:880)
                   at sun.reflect.GeneratedMethodAccessor100.invoke(Unknown Source)
                   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   at java.lang.reflect.Method.invoke(Method.java:597)
                   at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
                   at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
                   at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
                   at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
                   at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
                   at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
                   at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
                   at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
                   at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
                   at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
                   at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
                   at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
                   at org.jboss.ejb.Container.invoke(Container.java:960)
                   at sun.reflect.GeneratedMethodAccessor95.invoke(Unknown Source)
                   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   at java.lang.reflect.Method.invoke(Method.java:597)
                   at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
                   at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
                   at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
                   at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
                   at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
                   at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
                   at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
                   at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
                   at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
                   at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
                   at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
                   at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)
                   at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
                   at $Proxy61.processEMTEvent(Unknown Source)
                   at genesis.cmms.bean.CMMSMDBBean.onMessage(CMMSMDBBean.java:99)
                   at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source)
                   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   at java.lang.reflect.Method.invoke(Method.java:597)
                   at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
                   at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495)
                   at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
                   at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116)
                   at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
                   at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
                   at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
                   at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
                   at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)
                   at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
                   at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
                   at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402)
                   at org.jboss.ejb.Container.invoke(Container.java:960)
                   at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:987)
                   at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1287)
                   at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:266)
                   at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)
                   at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
                   at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect428.invoke(SessionAspect428.java)
                   at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                   at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                   at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                   at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                   at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                   at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
                   at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
                   at java.lang.Thread.run(Thread.java:619)
              

               

              I tried setting the remoting-bisocket-service.xml as below (testing mingjun jiang's theory), but somehow it keeps on occuring

               

              <?xml version="1.0" encoding="UTF-8"?>
              
              <server>
              
                 <!-- Standard bisocket connector - the bisocket transport only opens connection from client->server
                      so can be used with firewalls where only outgoing connections are allowed.
                      For examples of HTTP and SSL transports see docs/examples -->
                 <mbean code="org.jboss.remoting.transport.Connector"
                        name="jboss.messaging:service=Connector,transport=bisocket"
                        display-name="Bisocket Transport Connector">
                    <attribute name="Configuration">
                       <config>
                          <invoker transport="bisocket">
                          
                             <!-- There should be no reason to change these parameters - warning!
                                  Changing them may stop JBoss Messaging working correctly -->            
                             <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
                             <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
                             <attribute name="dataType" isParam="true">jms</attribute>
                             <attribute name="socket.check_connection" isParam="true">false</attribute>               
                             <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
                             <attribute name="serverBindPort">4457</attribute>
                             <attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
                             <attribute name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>
                          
                             <attribute name="numberOfCallRetries" isParam="true">1</attribute>
                             <attribute name="pingFrequency" isParam="true">214748364</attribute>
                             <attribute name="pingWindowFactor" isParam="true">10</attribute>
                             <attribute name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>               
                             <!-- End immutable parameters -->
                             
                             <attribute name="stopLeaseOnFailure" isParam="true">true</attribute>
                             
                             <!-- Periodicity of client pings. Server window by default is twice this figure -->                               
                             <attribute name="clientLeasePeriod" isParam="true">10000</attribute>
                             <attribute name="validatorPingPeriod" isParam="true">10000</attribute>
                             <attribute name="validatorPingTimeout" isParam="true">5000</attribute>
              
                             <attribute name="failureDisconnectTimeout" isParam="true">0</attribute>
                             <attribute name="callbackErrorsAllowed">1</attribute>
                             <attribute name="registerCallbackListener">false</attribute>
                             <attribute name="useClientConnectionIdentity" isParam="true">true</attribute>
                                      
                               <attribute name="timeout" isParam="true">10</attribute>
              
                             <!-- Number of seconds to wait for a connection in the client pool to become free -->
                             <attribute name="numberOfRetries" isParam="true">10</attribute>
              
                             <!-- Max Number of connections in client pool. This should be significantly higher than
                                  the max number of sessions/consumers you expect -->
                             <attribute name="JBM_clientMaxPoolSize" isParam="true">200</attribute>
                             
                             <!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->
                             <attribute name="callbackTimeout">10000</attribute> 
                             
                             <!-- Use these parameters to specify values for binding and connecting control connections to 
                                  work with your firewall/NAT configuration
                             <attribute name="secondaryBindPort">xyz</attribute>                           
                             <attribute name="secondaryConnectPort">abc</attribute>               
                             -->
                                        
                          </invoker>
                          <handlers>
                             <handler subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>
                          </handlers>
                       </config>
                    </attribute>
                 </mbean>
              
              </server>
              

               

              Am I doing it wrong?

              • 4. Re: Client hangs when get JMS connection factory
                Rakesh Naidu Newbie
                Has anyone found a workaround for this issue so far?
                • 5. Re: Client hangs when get JMS connection factory
                  mingjun jiang Newbie

                  Regarding this issue, I also opened a JIRA issue onto JBoss JIRA website: https://jira.jboss.org/jira/browse/JBREM-1169. One of JBoss expert commented on my issue, he just explained why this issue would happen, but not provided any workaround, I still think it should be bug........

                   

                  Finally, we degrade JBM to 1.4.2 GA and use JBoss Remoting jar 2.4.0 SP1 to overcome this problem. After applying JBM 1.4.2 GA and Remoting 2.4.0 SP1. Please refer to the link above for more details

                  • 6. Re: Client hangs when get JMS connection factory
                    Ron Sigal Master
                    Hi guys,

                    I'm going to talk a little about how ConnectionValidator works, which might help to clear up some confusion. 

                    Maybe the fact that "connection" can mean different things in different contexts is the source of some confusion.  A connection could be represented, for example, by a pair of sockets.  It is in that sense that a Remoting client invoker maintains a "connection pool".  It is, essentially, a set of sockets, each connected to a socket at the other end of the connection.  But, to an application like JBossMessaging (JBM), a connection is something at a higher level.  A JBM connection is associated with a Remoting client invoker, which manages a set of lower level connections.

                    Now, Remoting's ConnectionValidator checks for the usability of a higher level connection.  It has its own client invoker, with its own lower level connection pool.  When it can't get a response from the server, it assumes that the network between the client and the server is broken and it tells any registered listeners, such as JBM, that, in effect, the higher level connection is broken.

                    When a ConnectionValidator tells JBM that the higher level connection is broken, then JBM will tear down (and possibly try to recreate) the connection, which includes shutting down the associated Remoting client invoker and its connection pool.  So, "validatorPingTimeout" determines how long a ConnectionValidator will wait until it considers the higher level connection to be broken.  "timeout" determines how long read() operations on the client and the server will wait before timing out.

                    I hope that helps to clarify things.

                    Now, if you pull the ethernet connection, ConnectionValidator should detect that network failure.  In fact, I believe it works, having tested it myself.  Li Lin and Mingjun Jiang, is there no evidence in the client log that ConnectionValidator is informing JBM about the connection failure?

                    By the way, the "timeout" value is interpreted as milliseconds.  I noticed a "timeout" set to "10" earlier in the thread.

                    -Ron
                    • 7. Re: Client hangs when get JMS connection factory
                      mingjun jiang Newbie

                      Ron, thanks for your explaination. I know the meaning of "validatorPingTimeout" and "timeout". In Li Lin's original post, he said he solved this problem by changing the value of "timeout", actually, the changing of timeout value doesn't solve our problem, the changing of timeout value incurred another issue: you know, if there is no message in topic/queue in server side, then the remote client (message listener)  won't receive any message, that mean the connection is idle, if the idle duration exceeds the "timeout" value, then the JBM will forcibly close the connection, so the remoting client won't receive any subsequent messages any more, it's a serious problem!

                       

                      Once more, as what I mentioned in this thread or https://jira.jboss.org/jira/browse/JBREM-1169, if we pull the physical ethernet connection, the connection Validator won't detect the failure and close the failure connection correspondingly. But if we killed the client (i.e. kill the process of message listener in "Task Manager" of Windows OS), the value of "currentClientPoolSize" will decrease to 0 immediately, that meants the Connectionvalidator could detect and close the failure connection under such situation.

                      Ron, do you mean your test result show the ConnectionValidator could detect and close the failure connection when pulling out the physical ethernet connection?  Are you using the same version of JBM and remoting jar mentioned in Li Lin's post?

                      • 8. Re: Client hangs when get JMS connection factory
                        Rakesh Naidu Newbie


                        Similarly, I've managed to simulate the error as below:

                         

                        1. Start a client application on a remote computer
                        2. Abruptly kill this client application (in this case,using the Task Manager)
                        3. Check the SERVER.LOG generated for this period


                        The moment the application is killed, the marshalling exception appears on the SERVER.LOG. I actually kept the SERVER.LOG scrolling for this experiment & the exception appears exactly when the application is killed. I think this closely resembles the action of disconnecting the network point, where from the server's view, the Client has 'disappeared'. When I check the NonDurableMessage count at this very moment, I can see a spike in the number but it lasts for a few seconds only before returning to 0. I assume all the messages were consumed or dealt with properly by JBREM.

                         

                        Under a mass-scale environment like the one I'm in (with an excess of 100 nodes to 1 server), its difficult to pin-point which client triggered this exception or how it happened. All we have in the end of the day is the exception in the SERVER.LOG & a whole bunch of unconsumed messages under NonDurableMessages. Is it possible to track back which node caused this issue?


                        At the moment, all I can do is restart the server every time my memory runs low due to the message pile-up.

                            • 9. Re: Client hangs when get JMS connection factory
                              mingjun jiang Newbie

                              Hello Rakesh.

                               

                              According to your description of test procedure, you are using the kill client appliction using Task Manager, instead of pulling the network cable. Just as what I said in previous post, the connectionValidator could work correctly when killing client application, but it cannot work well when pulling network cable, the two test result (killing client application via Task manager  vs. pulling network cable) is different!  Could you test the scenario of pulling network cable?

                               

                              Regards

                              • 10. Re: Client hangs when get JMS connection factory
                                Ron Sigal Master

                                Hi guys,

                                 

                                I've done some experimentation and it looks like we're both right.  I was right that ConnectionValidator correctly detects and reports a connection failure when the ethernet cable is pulled.  That should be true for any recent version of Remoting, either 2.2.x or 2.5.x.  When ConnectionValidator tells JBossMessaging about the failure, JBM should close the connection, which should lead to a call to MicroSocketClientInvoker.handleDisconnect() in Remoting.  Now, MicroSocketClientInvoker.handleDisconnect() closes all inactive sockets, but it doesn't close sockets that are currently involved in an invocation.  That's why the number of sockets doesn't go to zero when "timeout" is set to 0.  But if "timeout" is set to a non-zero value, these active sockets will time out and get closed.

                                 

                                By the way, there has been some discussion about changing the default "timeout" configuration in remoting-bisocket-service.xml to a non-zero value.

                                 

                                -Ron

                                • 11. Re: Client hangs when get JMS connection factory
                                  Ron Sigal Master

                                  mjjiangbhr wrote:

                                   

                                  actually, the changing of timeout value doesn't solve our problem, the changing of timeout value incurred another issue: you know, if there is no message in topic/queue in server side, then the remote client (message listener)  won't receive any message, that mean the connection is idle, if the idle duration exceeds the "timeout" value, then the JBM will forcibly close the connection, so the remoting client won't receive any subsequent messages any more, it's a serious problem!

                                   

                                   

                                  I don't know enough about JBossMessaging to comment.  Can you elaborate?

                                   

                                  -Ron

                                  • 12. Re: Client hangs when get JMS connection factory
                                    mingjun jiang Newbie

                                    mjjiangbhr 编写:

                                     

                                    Hello Rakesh.

                                     

                                    According to your description of test procedure, you are using the kill client appliction using Task Manager, instead of pulling the network cable. Just as what I said in previous post, the connectionValidator could work correctly when killing client application, but it cannot work well when pulling network cable, the two test result (killing client application via Task manager  vs. pulling network cable) is different!  Could you test the scenario of pulling network cable?

                                     

                                    Regards

                                    Oops,Rakesh, please ignore my message above. I made a mistake, I wrongly treat you as Ron, sorry

                                    • 13. Re: Client hangs when get JMS connection factory
                                      mingjun jiang Newbie

                                      ron.sigal@jboss.com 编写:

                                       

                                      Hi guys,

                                       

                                      I've done some experimentation and it looks like we're both right.  I was right that ConnectionValidator correctly detects and reports a connection failure when the ethernet cable is pulled.  That should be true for any recent version of Remoting, either 2.2.x or 2.5.x.  When ConnectionValidator tells JBossMessaging about the failure, JBM should close the connection, which should lead to a call to MicroSocketClientInvoker.handleDisconnect() in Remoting.  Now, MicroSocketClientInvoker.handleDisconnect() closes all inactive sockets, but it doesn't close sockets that are currently involved in an invocation.  That's why the number of sockets doesn't go to zero when "timeout" is set to 0.  But if "timeout" is set to a non-zero value, these active sockets will time out and get closed.

                                       

                                      By the way, there has been some discussion about changing the default "timeout" configuration in remoting-bisocket-service.xml to a non-zero value.

                                       

                                      -Ron

                                      Ron, yes, in the original post, my colleague Li Lin had already said a non-zero timeout value will close the failure sockets. If you keep the timeout value as 0 (default value), the number of sockets won't go to zero. But if we change the timeout to a non-zero value, it will cause other issue just what I said previously

                                      • 14. Re: Client hangs when get JMS connection factory
                                        mingjun jiang Newbie

                                        ron.sigal@jboss.com 编写:

                                         


                                        mjjiangbhr wrote:

                                         

                                        actually, the changing of timeout value doesn't solve our problem, the changing of timeout value incurred another issue: you know, if there is no message in topic/queue in server side, then the remote client (message listener)  won't receive any message, that mean the connection is idle, if the idle duration exceeds the "timeout" value, then the JBM will forcibly close the connection, so the remoting client won't receive any subsequent messages any more, it's a serious problem!

                                         

                                         

                                        I don't know enough about JBossMessaging to comment.  Can you elaborate?

                                         

                                        -Ron

                                         

                                        Ron, please refer to this link: https://jira.jboss.org/jira/browse/JBREM-1169. Actually, you had already commented on this issue in the JBREM-1169

                                         

                                        In our application, there are a lot of Message listeners running on the client side, these message listeners will receive messages from queue/topic deployed in JBoss Messaging.

                                         

                                        If we set the timeout value to a non-zero value, and if there is no message in topic/queue in server side, then the remote client (message listener)  won't receive any message, that mean the connection between client and server is idle, if the idle duration exceeds the non-zero "timeout" value, then the JBM will forcibly close the connection, so the remoting client won't receive any subsequent messages from server any more......

                                        1 2 Previous Next