10 Replies Latest reply on Dec 2, 2008 12:06 AM by lanceliao1

    JBM deadlock cause by socket write

      Hi JBMGroup.
      We got a trouble about JBM.Using Jboss 4.22, JBM 1.4 sp3 cp04.After Long time run and load increase,the JBM Server lose any response.
      part thread dump:

      "WorkerThread#160[192.166.32.8:33764]" prio=3 tid=0x01664d88 nid=0x4d00 waiting for monitor entry [0x4927f000..0x4927fc70]
       at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:234)
       - waiting to lock <0x796c3d18> (a java.lang.Object)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
       at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
       at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
       at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
       at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
      
       "WorkerThread#159[192.166.32.8:33676]" prio=3 tid=0x0242f938 nid=0x4cdd runnable [0x4667e000..0x4667f9f0]
       at java.net.SocketOutputStream.socketWrite0(Native Method)
       at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
       at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
       at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
       at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
       - locked <0x7af9d5c0> (a java.io.BufferedOutputStream)
       at java.io.DataOutputStream.flush(DataOutputStream.java:106)
       at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:93)
       at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)
       at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:945)
       at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:586)
       at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
       at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
       at org.jboss.remoting.Client.invoke(Client.java:1634)
       at org.jboss.remoting.Client.invoke(Client.java:548)
       at org.jboss.remoting.Client.invokeOneway(Client.java:598)
       at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
       at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
       - locked <0x794462d8> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
       at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
       - locked <0x7a778760> (a java.lang.Object)
       at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
       at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
       at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
       at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
       at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
       at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)
       - locked <0x796c3d18> (a java.lang.Object)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
       at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
       at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
       at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
       at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
      
       "WorkerThread#158[192.166.32.8:33596]" prio=3 tid=0x010b6770 nid=0x4c90 waiting for monitor entry [0x4f67f000..0x4f67fc70]
       at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:234)
       - waiting to lock <0x796c3d18> (a java.lang.Object)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
       at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
       at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
       at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
       at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
      
      "WorkerThread#157[192.166.32.8:33731]" prio=3 tid=0x01c23f78 nid=0x4c78 waiting for monitor entry [0x4967f000..0x4967fc70]
       at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:234)
       - waiting to lock <0x796c3d18> (a java.lang.Object)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
       at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
       at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
       at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
       at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
      
      "WorkerThread#53[192.166.32.8:33606]" prio=3 tid=0x02497750 nid=0x281 waiting for monitor entry [0x5757f000..0x5757fbf0]
       at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:234)
       - waiting to lock <0x796c3d18> (a java.lang.Object)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
       at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
       at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
       at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
       at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
      
       "Thread-117" prio=3 tid=0x00f0b818 nid=0x21c waiting for monitor entry [0x4fe7f000..0x4fe7fa70]
       at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:350)
       - waiting to lock <0x796c3d18> (a java.lang.Object)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1539)
       at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
       at java.lang.Thread.run(Thread.java:595)

      All Thread are waiting to lock 0x796c3d18,but the lock holder are waiting to socket write.

      "WorkerThread#159[192.166.32.8:33676]" prio=3 tid=0x0242f938 nid=0x4cdd runnable [0x4667e000..0x4667f9f0]
       at java.net.SocketOutputStream.socketWrite0(Native Method)
       at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
       at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
       at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
       at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
       - locked <0x7af9d5c0> (a java.io.BufferedOutputStream)
       at java.io.DataOutputStream.flush(DataOutputStream.java:106)
       at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:93)
       at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)
       at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:945)
       at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:586)
       at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
       at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
       at org.jboss.remoting.Client.invoke(Client.java:1634)
       at org.jboss.remoting.Client.invoke(Client.java:548)
       at org.jboss.remoting.Client.invokeOneway(Client.java:598)
       at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
       at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
       - locked <0x794462d8> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
       at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
       - locked <0x7a778760> (a java.lang.Object)
       at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
       at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
       at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
       at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
       at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
       at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)
       - locked <0x796c3d18> (a java.lang.Object)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)

      the socket write thread infinite writing,can not return.
      we use netstat,found the port 4557 tcp recieve window is zero.
      192.166.32.8.4557 192.166.32.21.2154 65535 0 0 0 ESTABLISHED
      192.166.32.8.4557 192.166.32.21.2152 65535 0 0 0 ESTABLISHED
      192.166.32.8.33676 192.166.32.8.4557 49152 0 49152 0 ESTABLISHED
      192.166.32.8.4557 192.166.32.8.33676 49152 0 49152 0 ESTABLISHED

      the port 33676's receive window seems normal,but due to local network address,the output of loopback info can be ignore(The answer of Sun Consult).

      The things are seem to be:
      the lock holder thread are writing to port 4557,but the receive window is zero,can not write. the other reader thread writing to a lock to read from 4557. Is the probem biSocket ?




      BioSocket,

        • 1. Re: JBM deadlock cause by socket write
          timfox

          What version of JBoss Remoting are you using?

          Did you make sure you're using the same version of remoting on both client and server sides?

          • 2. Re: JBM deadlock cause by socket write

            Thanks for reply.
            We are using Remoting 2.2.2sp7. I make sure using thre same version of remoting on both client and server.The remoting-bisocket-service.xml file:

             <invoker transport="bisocket">
            
             <!-- There should be no reason to change these parameters - warning!
             Changing them may stop JBoss Messaging working correctly -->
             <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
             <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
             <attribute name="dataType" isParam="true">jms</attribute>
             <attribute name="socket.check_connection" isParam="true">false</attribute>
             <attribute name="timeout" isParam="true">0</attribute>
             <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
             <attribute name="serverBindPort">4457</attribute>
             <attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
             <attribute name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>
             <attribute name="numberOfCallRetries" isParam="true">1</attribute>
             <attribute name="pingFrequency" isParam="true">214748364</attribute>
             <attribute name="pingWindowFactor" isParam="true">10</attribute>
             <attribute name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>
             <!-- End immutable parameters -->
            
             <!-- Periodicity of client pings. Server window by default is twice this figure -->
             <attribute name="clientLeasePeriod" isParam="true">10000</attribute>
            
             <!-- Number of seconds to wait for a connection in the client pool to become free -->
             <attribute name="numberOfRetries" isParam="true">10</attribute>
            
             <!-- Max Number of connections in client pool. This should be significantly higher than
             the max number of sessions/consumers you expect -->
             <attribute name="JBM_clientMaxPoolSize" isParam="true">1000</attribute>
             <!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->
             <attribute name="callbackTimeout">10000</attribute>


            • 3. Re: JBM deadlock cause by socket write

              1 The MessageingPostOffice need a readwritelock to handle msg,such as route,binding,removeBinding.
              2 Once find a route,get readwritelock,it is using MessagingQueue to handle the msg(Delivery handle(DeliveryObserver observer, MessageReference reference, Transaction tx)).
              3 while handdling msg,the Correspond MessagingQueue require a lock to deliver msg.
              4 Then using RoundRobin policy choose one ServerSessionEndpoint to deliver msg by Remoting's bisocket.The serverSeeionEndpoint synchronize the method of handleDelivery.


              The question is that,one thread hung at step 4 due to no tcp buffer(4557 port recv window is zero) to write bytes.The other thead need a lock to process bytes on port 4557.It is the reason of deadlock,if not using bisocket may resolve the problem?

              • 4. Re: JBM deadlock cause by socket write
                gaohoward

                This is more like a tcp issue. How do you acknowledge messages on the consumer end?

                • 5. Re: JBM deadlock cause by socket write

                  Using auto_acknowledage mode create connection both on client and server side.
                  The similar codes are blow:

                  TopicConnectionFactory topicFactory = (TopicConnectionFactory) ContextUtil.getContext() .lookup("ConnectionFactory");
                  topicConn = topicFactory.createTopicConnection();
                  //set connection exceptionlistener
                  topicConn.setExceptionListener(new ExceptionListenerImpl());
                  
                  topicSession = topicConn.createTopicSession(false,Session.AUTO_ACKNOWLEDGE);
                  
                  channels = MsgRouterController.getInstance().findAllChannel();
                  
                  topicPublishers = new TopicPublisher[channels.length];
                  for (int i = 0; i < topicPublishers.length; i++) {
                   Topic topic = (Topic) ContextUtil.getContext().lookup(Const.Topic_Prefix+ channels);
                   topicPublishers = topicSession.createPublisher(topic);
                   log.debug("topicPublisher "+channels+" created!");
                   }


                  if there is one client block receive jms due to network problem,I think the server should not block all receivers and senders.
                  My problem is waiting on socketwrite(),and blocking others thread to receive msg or send msg.


                  • 6. Re: JBM deadlock cause by socket write

                    The memory dump,I hope it will helpful.
                    Why 287 serversessionEndpoint and 133 ServerConsumerEndpoint? We have about 10 client_end receive and send msg,each end have 19 topic.

                    2124 152928 org.jboss.jms.message.JBossObjectMessage
                    2277 54648 org.jboss.jms.destination.JBossTopic
                     287 36736 org.jboss.jms.server.endpoint.ServerSessionEndpoint
                     133 11704 org.jboss.jms.server.endpoint.ServerConsumerEndpoint
                     205 8200 org.jboss.jms.server.container.SecurityAspect
                     135 7560 org.jboss.jms.server.messagecounter.MessageCounter
                     64 6656 org.jboss.jms.server.endpoint.ServerConnectionEndpoint
                     196 6272 org.jboss.jms.server.remoting.ServerSocketWrapper
                     189 4536 org.jboss.jms.server.JMSCondition
                     282 4512 org.jboss.jms.server.endpoint.advised.SessionAdvised
                     48 4224 org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891
                     135 2160 org.jboss.jms.server.messagecounter.MessageCounter$DayCounter
                     133 2128 org.jboss.jms.server.endpoint.advised.ConsumerAdvised
                     48 1920 org.jboss.jms.wireformat.SessionSendRequest
                     47 1880 org.jboss.jms.wireformat.ClientDelivery
                     115 1840 org.jboss.jms.wireformat.JMSWireFormat
                     21 1680 org.jboss.jms.client.state.SessionState
                     19 1368 org.jboss.jms.server.destination.ManagedTopic
                     19 1216 org.jboss.jms.server.destination.TopicService
                     49 1176 org.jboss.jms.client.container.ClosedInterceptor
                     11 1056 org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886
                     64 1024 org.jboss.jms.server.endpoint.advised.ConnectionAdvised
                     11 880 org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588
                     18 864 org.jboss.jms.client.state.ProducerState
                     21 840 org.jboss.jms.client.delegate.ClientSessionDelegate
                     49 784 org.jboss.jms.client.container.DelegateIdentity
                     31 744 org.jboss.jms.client.remoting.ClientSocketWrapper
                     17 680 org.jboss.jms.asf.StdServerSession
                     13 624 org.jboss.jms.message.ObjectMessageProxy
                     18 576 org.jboss.jms.client.delegate.ClientProducerDelegate
                     31 496 org.jboss.jms.client.container.FailoverValveInterceptor
                     7 448 org.jboss.jms.client.state.ConnectionState
                     17 408 org.jboss.jms.tx.MessagingXAResource
                     17 408 org.jboss.jms.tx.ClientTransaction
                     7 336 org.jboss.jms.client.delegate.ClientConnectionDelegate
                     3 288 org.jboss.jms.client.container.ClientConsumer
                     18 288 org.jboss.jms.client.JBossSession
                     3 288 org.jboss.jms.server.connectionfactory.ConnectionFactory
                     18 288 org.jboss.jms.client.JBossMessageProducer
                     15 240 org.jboss.jms.tx.LocalTx
                     3 240 org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585
                     1 232 org.jboss.jms.server.ServerPeer
                     7 224 org.jboss.jms.client.remoting.JMSRemotingConnection
                     3 192 org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint
                     3 192 org.jboss.jms.asf.StdServerSessionPool
                     3 192 org.jboss.jms.client.state.ConsumerState
                     7 168 org.jboss.jms.client.container.ConnectionAspect
                     7 168 org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener
                     3 168 org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate
                     2 160 org.jboss.jms.server.destination.ManagedQueue
                     3 144 org.jboss.jms.asf.StdServerSessionPool$MyPooledExecutor
                     3 144 org.jboss.jms.client.delegate.ClientConsumerDelegate
                     3 144 org.jboss.jms.client.JBossConnectionConsumer
                     2 144 org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702
                     2 128 org.jboss.jms.server.destination.QueueService
                     7 112 org.jboss.jms.client.remoting.CallbackManager
                     7 112 org.jboss.jms.client.JBossConnection
                     1 72 org.jboss.jms.server.endpoint.advised.ConsumerAdvised$close_N4742752445160157748
                     1 72 org.jboss.jms.server.plugin.JDBCJMSUserManagerService
                     1 72 org.jboss.jms.jndi.JMSProviderLoader
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect34
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect40
                     2 64 org.jboss.jms.delegate.DeliveryInfo
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect33
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect55
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect35
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect19
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect3
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect54
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect41
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect36
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect4
                     4 64 org.jboss.jms.server.security.SecurityMetadata$Role
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect21
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect18
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect37
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect1
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect5
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect53
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect42
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28
                     2 64 org.jboss.jms.tx.MessagingXid
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect32
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect38
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect52
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect17
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect6
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect2
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect47
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect43
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect39
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect51
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect22
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect7
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect27
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect16
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect31
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect50
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect44
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect23
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect15
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect8
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect45
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect9
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect26
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect49
                     2 64 org.jboss.jms.tx.ClientTransaction$SessionTxState
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect30
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect46
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect24
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect10
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect29
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect12
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect13
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect25
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect20
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect11
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect48
                     1 64 org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect56
                     1 56 org.jboss.jms.asf.ServerSessionPoolLoader
                     3 48 org.jboss.jms.server.security.CheckType
                     3 48 org.jboss.jms.server.endpoint.advised.ConnectionFactoryAdvised
                     2 48 org.jboss.jms.destination.JBossQueue
                     1 40 org.jboss.jms.server.Bootstrap
                     1 40 org.jboss.jms.server.messagecounter.MessageCounterManager
                     1 40 org.jboss.jms.server.security.SecurityMetadataStore
                     1 32 org.jboss.jms.jndi.JNDIProviderAdapter
                     1 32 org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper
                     1 32 org.jboss.jms.server.connectionmanager.SimpleConnectionManager
                     1 32 org.jboss.jms.server.plugin.JDBCJMSUserManager
                     1 32 org.jboss.jms.message.MessageIdGenerator
                     2 32 org.jboss.jms.server.connectionfactory.JNDIBindings
                     2 32 org.jboss.jms.client.JBossConnectionFactory
                     4 32 org.jboss.jms.server.remoting.DirectThreadPool
                     1 24 org.jboss.jms.server.remoting.JMSServerInvocationHandler
                     1 24 org.jboss.jms.tx.ResourceManager
                     3 24 org.jboss.jms.asf.StdServerSessionPool$DefaultThreadFactory
                     1 24 org.jboss.jms.asf.StdServerSessionPoolFactory
                     1 24 org.jboss.jms.server.security.SecurityMetadata
                     1 24 org.jboss.jms.server.DestinationJNDIMapper
                     1 24 org.jboss.jms.message.MessageIdGeneratorFactory$Holder
                     1 24 org.jboss.jms.wireformat.CloseRequest
                     1 16 org.jboss.jms.wireformat.Dispatcher
                     1 16 org.jboss.jms.message.MessageIdGeneratorFactory
                     1 16 org.jboss.jms.server.container.ServerLogInterceptor
                     1 16 org.jboss.jms.client.ClientAOPStackLoader
                     1 16 org.jboss.jms.wireformat.SerializedPacket
                     1 16 org.jboss.jms.client.container.StateCreationAspect
                     1 16 org.jboss.jms.client.container.ProducerAspect
                     1 16 org.jboss.jms.client.container.SessionAspect
                     1 16 org.jboss.jms.tx.ResourceManagerFactory
                     1 16 org.jboss.jms.tx.ResourceManagerFactory$Holder
                     1 16 org.jboss.jms.server.connectormanager.SimpleConnectorManager
                     1 16 org.jboss.jms.server.endpoint.ServerSessionEndpoint$2
                     1 8 org.jboss.jms.server.container.SecurityActions$PrincipalInfoAction$1
                     1 8 org.jboss.jms.server.JMSConditionFactory
                     1 8 org.jboss.jms.server.selector.SelectorFactory
                     1 8 org.jboss.jms.client.container.ConsumerAspect
                     1 8 org.jboss.jms.server.endpoint.SecurityActions$PrincipalInfoAction$1
                     1 8 org.jboss.jms.server.security.SecurityActions$PrincipalInfoAction$2
                     1 8 org.jboss.jms.server.endpoint.SecurityActions$PrincipalInfoAction$2
                     1 8 org.jboss.jms.server.security.SecurityActions$PrincipalInfoAction$1
                     1 8 org.jboss.jms.client.plugin.RoundRobinLoadBalancingFactory
                     1 8 org.jboss.jms.server.container.SecurityActions$PrincipalInfoAction$2
                    


                    • 7. Re: JBM deadlock cause by socket write

                      Socket info
                      Solaris: netstat -an
                      Local Address Remote Address Swind Send-Q Rwind Recv-Q State
                      -------------------- -------------------- ----- ------ ----- ------ -----------
                      192.166.32.8.4557 192.166.32.155.52066 64679 0 49640 0 ESTABLISHED
                      192.166.32.8.4557 192.166.32.180.3199 65535 0 49640 0 ESTABLISHED
                      192.166.32.8.4557 192.166.32.96.1601 65535 0 0 0 ESTABLISHED
                      192.166.32.8.56726 192.166.32.8.4557 49152 0 49152 0 ESTABLISHED
                      192.166.32.8.4557 192.166.32.244.4245 65535 0 49640 0 ESTABLISHED
                      192.166.32.8.4557 192.166.32.155.52297 64606 0 49640 0 ESTABLISHED
                      192.166.32.8.4557 192.166.88.88.3746 65520 0 49640 0 ESTABLISHED
                      192.166.32.8.4557 192.166.88.88.1593 64647 0 49640 0 ESTABLISHED
                      192.166.32.8.4557 192.166.32.21.1872 65535 0 0 0 ESTABLISHED
                      192.166.32.8.4557 192.166.32.100.7408 64196 0 49640 0 ESTABLISHED
                      192.166.32.8.4557 192.166.32.180.2108 64149 0 49640 0 ESTABLISHED
                      192.166.32.8.4557 192.166.32.21.2152 65535 0 0 0 ESTABLISHED
                      192.166.32.8.4557 192.166.32.21.2154 65535 0 0 0 ESTABLISHED
                      ... more than 200 line.

                      The following lines Rwind are zero,meaning the socketwrite() kept waiting(more than 2 hours,never return),the correspond reader thread does not working.
                      192.166.32.8.4557 192.166.32.96.1601
                      192.166.32.8.4557 192.166.32.21.1872
                      192.166.32.8.4557 192.166.32.21.2152
                      192.166.32.8.4557 192.166.32.21.2154

                      ***************************************************************************************************************************
                      Thread dump(full dump are too large,only select some)

                      "WorkerThread#47[192.166.32.96:1601]" prio=3 tid=0x0139eb68 nid=0x264 in Object.wait() [0x4be7f000..0x4be7fa70]
                       at java.lang.Object.wait(Native Method)
                       at java.lang.Object.wait(Object.java:474)
                       at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
                       - locked <0x7964b110> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
                       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2075)
                       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
                       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
                       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                       at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
                       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                       at java.lang.reflect.Method.invoke(Method.java:585)
                       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                       at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
                       at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
                       at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
                       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
                       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                      
                      "WorkerThread#36[192.166.32.21:1872]" prio=3 tid=0x00fc1a00 nid=0x23f in Object.wait() [0x4de7f000..0x4de7faf0]
                       at java.lang.Object.wait(Native Method)
                       at java.lang.Object.wait(Object.java:474)
                       at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
                       - locked <0x7964b110> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
                       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2075)
                       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
                       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
                       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                       at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
                       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                       at java.lang.reflect.Method.invoke(Method.java:585)
                       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                       at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
                       at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
                       at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
                       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
                       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                      
                      "WorkerThread#164[192.166.32.21:2152]" prio=3 tid=0x02862648 nid=0x4d22 in Object.wait() [0x4897f000..0x4897f970]
                       at java.lang.Object.wait(Native Method)
                       at java.lang.Object.wait(Object.java:474)
                       at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
                       - locked <0x7964b110> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
                       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2075)
                       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
                       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
                       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                       at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
                       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                       at java.lang.reflect.Method.invoke(Method.java:585)
                       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                       at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
                       at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
                       at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
                       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
                       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                      
                      "WorkerThread#165[192.166.32.21:2154]" prio=3 tid=0x0383c480 nid=0x4d24 in Object.wait() [0x4947f000..0x4947fa70]
                       at java.lang.Object.wait(Native Method)
                       at java.lang.Object.wait(Object.java:474)
                       at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
                       - locked <0x7964b110> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
                       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2075)
                       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
                       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
                       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                       at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
                       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                       at java.lang.reflect.Method.invoke(Method.java:585)
                       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                       at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
                       at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
                       at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
                       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
                       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                      

                      ********************************************************************************************************************************

                      All these thread are waiting on MessagingPostOffice.routInternal(2075),waiting a readwrite lock.
                      lock.readLock().acquire();

                      the lock holder is:
                      "WorkerThread#159[192.166.32.8:33676]" prio=3 tid=0x0242f938 nid=0x4cdd runnable [0x4667e000..0x4667f9f0]
                       at java.net.SocketOutputStream.socketWrite0(Native Method)
                       at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
                       at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
                       at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                       at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                       - locked <0x7af9d5c0> (a java.io.BufferedOutputStream)
                       at java.io.DataOutputStream.flush(DataOutputStream.java:106)
                       at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:93)
                       at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)
                       at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:945)
                       at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:586)
                       at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
                       at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
                       at org.jboss.remoting.Client.invoke(Client.java:1634)
                       at org.jboss.remoting.Client.invoke(Client.java:548)
                       at org.jboss.remoting.Client.invokeOneway(Client.java:598)
                       at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
                       at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
                       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
                       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
                       - locked <0x794462d8> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
                       at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
                       - locked <0x7a778760> (a java.lang.Object)
                       at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
                       at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
                       at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
                       at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
                       at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
                       at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)
                       - locked <0x796c3d18> (a java.lang.Object)
                       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
                       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
                       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
                       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                       at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
                       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                       at java.lang.reflect.Method.invoke(Method.java:585)
                       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                       at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
                       at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
                       at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
                       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
                       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                      
                      "pool-1-thread-2" prio=3 tid=0x01efd3f0 nid=0x274 waiting for monitor entry [0x4af7e000..0x4af7fa70]
                       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1299)
                       - waiting to lock <0x794462d8> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
                       at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
                       - locked <0x7972c618> (a java.lang.Object)
                       at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
                       at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
                       at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
                       at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
                       at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
                       at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)
                       - locked <0x79730070> (a java.lang.Object)
                       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
                       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
                       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:748)
                       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                       at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
                       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                       at java.lang.reflect.Method.invoke(Method.java:585)
                       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                       at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
                       at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
                       at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
                       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
                       at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
                       at org.jboss.remoting.Client.invoke(Client.java:1634)
                       at org.jboss.remoting.Client.invoke(Client.java:548)
                       at org.jboss.remoting.Client.invokeOneway(Client.java:598)
                       at org.jboss.remoting.Client.invokeOneway(Client.java:786)
                       at org.jboss.remoting.Client.invokeOneway(Client.java:776)
                       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180)
                       at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165)
                       at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:473)
                       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                       at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
                       at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
                       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
                       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                       at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
                       at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
                       at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect46.invoke(ProducerAspect46.java)
                       at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                       at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                       at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
                       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
                       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
                       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
                       at org.jboss.jms.client.JBossMessageProducer.publish(JBossMessageProducer.java:188)
                       at com.datangmobile.oss.platform.jms.server.JmsSender.send(JmsSender.java:193)
                       at com.datangmobile.oss.omc.server.cm.mmf.sliptree.SlipMitTree.bstRequestMO(SlipMitTree.java:337)
                       at com.datangmobile.oss.omc.server.cm.mmf.sliptree.SlipMitTree.requestDynamicMO(SlipMitTree.java:513)
                       at com.datangmobile.oss.omc.server.cm.mmf.sliptree.SlipNeTree.requestDynamicMO(SlipNeTree.java:207)
                       at com.datangmobile.oss.omc.server.cm.ejb.SvrConfigMDB.neDynamicChange(SvrConfigMDB.java:255)
                       at com.datangmobile.oss.omc.server.cm.ejb.SvrConfigMDB.processMessageWithThread(SvrConfigMDB.java:163)
                       at com.datangmobile.oss.omc.server.cm.ejb.SvrConfigMDB$1.processMessage(SvrConfigMDB.java:147)
                       at com.datangmobile.oss.platform.jms.msg.dispatch.AbstractProcessor.processMessages(AbstractProcessor.java:24)
                       at com.datangmobile.oss.platform.jms.msg.dispatch.ProcessMsgWorker.run(MsgDispatcher.java:187)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
                       at java.lang.Thread.run(Thread.java:595)
                      

                      at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203),The thread got MessagingPostOffice's readwrite lock to route msg,and it locked MessagingQueue instance to deliver msg.
                      Unfortunately,the thread is waiting on socketwrite() forever(due to no tcp receive windows)
                      And one thread which holder MessagingQueue - locked <0x79730070> (a java.lang.Object) is waiting to lock <0x794462d8> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint) which holded by previous thread.
                      ********************************************************************************************************************************
                      The delivey(post msg to subscriber or recevie msg form publisher) threads in JBM server are waiting to MessagingPostOffice's readwrite lock.
                      The threads can not process any msg due to waiting to lock.So the tcp receive window is full,and deadlock happens.


                      • 8. Re: JBM deadlock cause by socket write
                        timfox
                        • 9. Re: JBM deadlock cause by socket write

                          Thanks a lot.
                          I have read the Issue.I make sure my remoting-bisocket-service.xml have the line and the remoting version is 2.2.2sp7:

                          <!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->
                           <attribute name="callbackTimeout">10000</attribute>
                          

                          I will try to set prefetchSize smaller to verify.
                          Why the
                          https://jira.jboss.org/jira/secure/attachment/12317850/stack_traces.zip can not download.I need stack trace to confirm there are the same problem.


                          • 10. Re: JBM deadlock cause by socket write

                            The problem has been resolved.