3 Replies Latest reply on May 6, 2004 6:18 AM by Adrian Brock

    deadlock in invocation layers on solaris

    null null Newbie

      Hello all,

      We are experiencing some annoying deadlock behavior when trying to send messages a queue. The code we use for sending a message is pretty straightforward:

       public static void sendRequestsOnQueue(QueueConnectionFactory factory, Queue queue, List requests, int deliveryMode) throws JMSException {
       QueueConnection queueConnection = factory.createQueueConnection();
       try {
       QueueSession qs = queueConnection.createQueueSession(false, javax.jms.Session.AUTO_ACKNOWLEDGE);
       try {
       QueueSender sender = qs.createSender(queue);
       try {
       sender.setDeliveryMode(deliveryMode);
       for (Iterator iterator = requests.iterator(); iterator.hasNext();) {
       Serializable request = (Serializable) iterator.next();
       sender.send(qs.createObjectMessage(request));
       }
       m_log.debug("Posted " + requests.size() + " jobs on queue " + queue.getQueueName());
       } finally {
       sender.close();
       }
       } finally {
       qs.close();
       }
       } finally {
       queueConnection.close();
       }
       }
      



      Deadlocks appear with both OIL and UIL2 invokers. I will add the threaddump for the OIL case below. Only the most important threads are show

      "OILClientILService-164" daemon prio=5 tid=0x814b48 nid=0x4ec runnable [e0b81000..e0b819bc]
       at java.net.SocketInputStream.socketRead0(Native Method)
       at java.net.SocketInputStream.read(SocketInputStream.java:129)
       at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
       at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
       - locked <e9334180> (a java.io.BufferedInputStream)
       at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2133)
       at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2316)
       at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2383)
       at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2455)
       at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2604)
       at java.io.ObjectInputStream.readByte(ObjectInputStream.java:845)
       at org.jboss.mq.il.oil.OILClientILService.run(OILClientILService.java:183)
       at java.lang.Thread.run(Thread.java:536)
      
      "OIL Worker-164" prio=5 tid=0xb1f828 nid=0x4e9 runnable [e1580000..e15819bc]
       at java.net.SocketInputStream.socketRead0(Native Method)
       at java.net.SocketInputStream.read(SocketInputStream.java:129)
       at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
       at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
       - locked <e9332618> (a java.io.BufferedInputStream)
       at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2133)
       at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2316)
       at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2383)
       at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2455)
       at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2604)
       at java.io.ObjectInputStream.readByte(ObjectInputStream.java:845)
       at org.jboss.mq.il.oil.OILClientIL.waitAnswer(OILClientIL.java:200)
       at org.jboss.mq.il.oil.OILClientIL.close(OILClientIL.java:74)
       - locked <e9331790> (a org.jboss.mq.il.oil.OILClientIL)
       at org.jboss.mq.server.JMSDestinationManager.connectionClosing(JMSDestinationManager.java:571)
       at org.jboss.mq.server.JMSServerInterceptorSupport.connectionClosing(JMSServerInterceptorSupport.java:113)
       at org.jboss.mq.security.ServerSecurityInterceptor.connectionClosing(ServerSecurityInterceptor.java:61)
       at org.jboss.mq.server.TracingInterceptor.connectionClosing(TracingInterceptor.java:185)
       at org.jboss.mq.server.JMSServerInvoker.connectionClosing(JMSServerInvoker.java:113)
       at org.jboss.mq.il.oil.OILServerILService$Client.run(OILServerILService.java:264)
       at java.lang.Thread.run(Thread.java:536)
      
      "Message Pushers-1" daemon prio=5 tid=0x10c0140 nid=0x54 in Object.wait() [e1281000..e12819bc]
       at java.lang.Object.wait(Native Method)
       - waiting on <e8e13ec0> (a java.util.LinkedList)
       at java.lang.Object.wait(Object.java:426)
       at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java:205)
       at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java:224)
       - locked <e8e13ec0> (a java.util.LinkedList)
      
      
      "Connection Monitor Thread" daemon prio=5 tid=0x10beeb0 nid=0x52 waiting for monitor entry [e1481000..e14819bc]
       at org.jboss.mq.il.oil.OILServerIL.ping(OILServerIL.java:421)
       - waiting to lock <e932d978> (a org.jboss.mq.il.oil.OILServerIL)
       at org.jboss.mq.Connection.pingServer(Connection.java:1141)
       at org.jboss.mq.Connection$PingTask.run(Connection.java:1307)
       at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
       at java.lang.Thread.run(Thread.java:536)
      
      "Thread-33" daemon prio=5 tid=0x10b42a0 nid=0x4e in Object.wait() [e1781000..e17819bc]
       at java.lang.Object.wait(Native Method)
       - waiting on <e8e62f20> (a org.jboss.mx.util.ThreadPool$Worker)
       at java.lang.Object.wait(Object.java:426)
       at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:243)
       - locked <e8e62f20> (a org.jboss.mx.util.ThreadPool$Worker)
      
      "Thread-32" daemon prio=5 tid=0xeca348 nid=0x4d runnable [e1880000..e18819bc]
       at java.net.SocketInputStream.socketRead0(Native Method)
       at java.net.SocketInputStream.read(SocketInputStream.java:129)
       at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
       at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
       - locked <e932f518> (a java.io.BufferedInputStream)
       at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2133)
       at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2316)
       at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2383)
       at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2455)
       at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2604)
       at java.io.ObjectInputStream.readByte(ObjectInputStream.java:845)
       at org.jboss.mq.il.oil.OILServerIL.waitAnswer(OILServerIL.java:601)
       at org.jboss.mq.il.oil.OILServerIL.connectionClosing(OILServerIL.java:334)
       - locked <e932d978> (a org.jboss.mq.il.oil.OILServerIL)
       at org.jboss.mq.Connection.close(Connection.java:491)
       - locked <e932d738> (a org.jboss.mq.SpyConnection)
       at com.alcatel.util.jms.JmsUtil.sendRequestsOnQueue(JmsUtil.java:63)
       at com.alcatel.ni.datacollection.collection.SchedulableCollector.sendAsamCollectionRequest(SchedulableCollector.java:292)
       at com.alcatel.ni.datacollection.collection.SchedulableCollector.collect(SchedulableCollector.java:218)
       at sun.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:324)
       at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
       at org.jboss.varia.scheduler.Scheduler$MBeanListener.handleNotification(Scheduler.java:1354)
       at org.jboss.mx.server.NotificationListenerProxy.handleNotification(NotificationListenerProxy.java:69)
       at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:95)
       at javax.management.timer.Timer.sendNotifications(Timer.java:441)
       at javax.management.timer.Timer.access$000(Timer.java:31)
       at javax.management.timer.Timer$RegisteredNotification.doRun(Timer.java:612)
       at org.jboss.mx.util.SchedulableRunnable.run(SchedulableRunnable.java:164)
       at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:225)
      
      


      Currently, we worked around the problem by using the JVM invoker and skipping to complete socket layer. Although it fits our needs currently, it will give problems the day we need to be distributed.

      We are using jboss 3.2.3 on solaris 2.8 running on a dual process E250. Another fact noting is that we were not able to reproduce this behavior on an NT box.

        • 1. Re: deadlock in invocation layers on solaris
          Adrian Brock Master

          Show me the thread dump for UIL2
          OIL is deprecated because it has known synchronization problems.

          Are the first two threads supposed to match up? If so they are both waiting for
          each other, nothing is going to move. I would need to see a log to work out
          why the server is waiting for a response from a client that is not doing anything.

          There have been problems reported before with Solaris because it has some
          stupid keep alive setting. I am sure you can find it using search.

          Finally, if these are in the same VM, why aren't you using java:/ConnectionFactory
          or java:/JmsXA?

          • 2. Re: deadlock in invocation layers on solaris
            null null Newbie

            Here's the full UIL2 thread dump:


            Full thread dump Java HotSpot(TM) Client VM (1.4.1_02-b06 mixed mode):
            
            "TP-Processor8" daemon prio=5 tid=0x11e50e0 nid=0x3ef7 runnable [e1381000..e13819bc]
             at java.net.PlainSocketImpl.socketAccept(Native Method)
             at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
             - locked <e7f69860> (a java.net.PlainSocketImpl)
             at java.net.ServerSocket.implAccept(ServerSocket.java:439)
             at java.net.ServerSocket.accept(ServerSocket.java:410)
             at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:107)
             at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:387)
             at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:569)
             at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
             at java.lang.Thread.run(Thread.java:536)
            
            "TP-Processor7" daemon prio=5 tid=0x11e4f48 nid=0x3ef6 in Object.wait() [e1481000..e14819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e93c3620> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Object.wait(Object.java:426)
             at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
             - locked <e93c3620> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Thread.run(Thread.java:536)
            
            "TP-Processor6" daemon prio=5 tid=0x11e2408 nid=0x3ef5 in Object.wait() [e1c81000..e1c819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e93c36a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Object.wait(Object.java:426)
             at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
             - locked <e93c36a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Thread.run(Thread.java:536)
            
            "TP-Processor5" daemon prio=5 tid=0xffc7b0 nid=0x3ef4 in Object.wait() [e3301000..e33019bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e93c3720> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Object.wait(Object.java:426)
             at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
             - locked <e93c3720> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Thread.run(Thread.java:536)
            
            "UIL2.SocketManager.WriteTask#3928" daemon prio=5 tid=0x1060140 nid=0x2f1e in Object.wait() [dea81000..dea819bc]
             at java.lang.Object.wait(Native Method)
             at java.lang.Object.wait(Object.java:426)
             at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
             - locked <e91eea98> (a java.lang.Object)
             at org.jboss.mq.il.uil2.SocketManager$WriteTask.run(SocketManager.java:473)
             at java.lang.Thread.run(Thread.java:536)
            
            "RMI TCP Connection(17)-172.31.249.2" daemon prio=5 tid=0xfd88f8 nid=0x25c4 runnable [df281000..df2819bc]
             at java.net.SocketInputStream.socketRead0(Native Method)
             at java.net.SocketInputStream.read(SocketInputStream.java:129)
             at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
             at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
             - locked <e9192f10> (a java.io.BufferedInputStream)
             at java.io.FilterInputStream.read(FilterInputStream.java:66)
             at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:442)
             at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
             at java.lang.Thread.run(Thread.java:536)
            
            "RMI LeaseChecker" daemon prio=5 tid=0xfd8760 nid=0x25c3 waiting on condition [dfb81000..dfb819bc]
             at java.lang.Thread.sleep(Native Method)
             at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:307)
             at java.lang.Thread.run(Thread.java:536)
            
            "JNP Server" prio=5 tid=0x10a9dc0 nid=0x25c1 runnable [dfd81000..dfd819bc]
             at java.net.PlainSocketImpl.socketAccept(Native Method)
             at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
             - locked <e7fed7b0> (a java.net.PlainSocketImpl)
             at java.net.ServerSocket.implAccept(ServerSocket.java:439)
             at java.net.ServerSocket.accept(ServerSocket.java:410)
             at org.jnp.server.Main.run(Main.java:281)
             at java.lang.Thread.run(Thread.java:536)
            
            "Thread-119" prio=5 tid=0x11573d0 nid=0xc1 in Object.wait() [e1581000..e15819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e8ece2d0> (a java.lang.Object)
             at java.lang.Object.wait(Object.java:426)
             at org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:772)
             - locked <e8ece2d0> (a java.lang.Object)
            
            "Message Pushers-1" daemon prio=5 tid=0xcb2530 nid=0x5c in Object.wait() [e0d81000..e0d819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e8d249f8> (a java.util.LinkedList)
             at java.lang.Object.wait(Object.java:426)
             at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java:205)
             at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java:224)
             - locked <e8d249f8> (a java.util.LinkedList)
            
            "StandardManager[]" daemon prio=5 tid=0xef3988 nid=0x5b waiting on condition [e0e81000..e0e819bc]
             at java.lang.Thread.sleep(Native Method)
             at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
             at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
             at java.lang.Thread.run(Thread.java:536)
            
            "Connection Monitor Thread" daemon prio=5 tid=0xad09a0 nid=0x59 in Object.wait() [e1081000..e10819bc]
             at java.lang.Object.wait(Native Method)
             at java.lang.Object.wait(Object.java:426)
             at EDU.oswego.cs.dl.util.concurrent.ClockDaemon.nextTask(ClockDaemon.java:321)
             - locked <e8d06568> (a EDU.oswego.cs.dl.util.concurrent.ClockDaemon)
             at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:362)
             at java.lang.Thread.run(Thread.java:536)
            
            "Thread-38" daemon prio=5 tid=0xf43fb8 nid=0x52 in Object.wait() [e1681000..e16819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e8df1610> (a org.jboss.mx.util.ThreadPool$Worker)
             at java.lang.Object.wait(Object.java:426)
             at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:243)
             - locked <e8df1610> (a org.jboss.mx.util.ThreadPool$Worker)
            
            "Thread-37" daemon prio=5 tid=0xdfcc58 nid=0x51 in Object.wait() [e1781000..e17819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e8df1668> (a org.jboss.mx.util.ThreadPool$Worker)
             at java.lang.Object.wait(Object.java:426)
             at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:243)
             - locked <e8df1668> (a org.jboss.mx.util.ThreadPool$Worker)
            
            "Thread-36" daemon prio=5 tid=0xf44fd8 nid=0x50 in Object.wait() [e1981000..e19819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e8df16c0> (a org.jboss.mx.util.ThreadPool$Worker)
             at java.lang.Object.wait(Object.java:426)
             at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:243)
             - locked <e8df16c0> (a org.jboss.mx.util.ThreadPool$Worker)
            
            "Thread-35" daemon prio=5 tid=0xacc000 nid=0x4f in Object.wait() [e1a81000..e1a819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e8df1718> (a org.jboss.mx.util.ThreadPool$Worker)
             at java.lang.Object.wait(Object.java:426)
             at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:243)
             - locked <e8df1718> (a org.jboss.mx.util.ThreadPool$Worker)
            
            "Thread-34" daemon prio=5 tid=0xf88ed0 nid=0x4e in Object.wait() [e1b80000..e1b819bc]
             at java.lang.Object.wait(Native Method)
             at java.lang.Object.wait(Object.java:426)
             at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:240)
             - locked <e91ef420> (a org.jboss.mq.il.uil2.msgs.CheckUserMsg)
             at org.jboss.mq.il.uil2.SocketManager.sendMessage(SocketManager.java:189)
             at org.jboss.mq.il.uil2.UILServerIL.authenticate(UILServerIL.java:302)
             at org.jboss.mq.Connection.authenticate(Connection.java:876)
             at org.jboss.mq.Connection.<init>(Connection.java:238)
             at org.jboss.mq.Connection.<init>(Connection.java:315)
             at org.jboss.mq.SpyConnection.<init>(SpyConnection.java:60)
             at org.jboss.mq.SpyConnectionFactory.createQueueConnection(SpyConnectionFactory.java:116)
             at com.alcatel.util.jms.JmsUtil.createQueueConnection(JmsUtil.java:28)
             at com.alcatel.util.jms.JmsUtil.sendRequestsOnQueue(JmsUtil.java:44)
             at com.alcatel.ni.datacollection.collection.SchedulableCollector.sendAsamCollectionRequest(SchedulableCollector.java:292)
             at com.alcatel.ni.datacollection.collection.SchedulableCollector.collect(SchedulableCollector.java:218)
             at sun.reflect.GeneratedMethodAccessor200.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:324)
             at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
             at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
             at org.jboss.varia.scheduler.Scheduler$MBeanListener.handleNotification(Scheduler.java:1354)
             at org.jboss.mx.server.NotificationListenerProxy.handleNotification(NotificationListenerProxy.java:69)
             at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:95)
             at javax.management.timer.Timer.sendNotifications(Timer.java:441)
             at javax.management.timer.Timer.access$000(Timer.java:31)
             at javax.management.timer.Timer$RegisteredNotification.doRun(Timer.java:612)
             at org.jboss.mx.util.SchedulableRunnable.run(SchedulableRunnable.java:164)
             at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:225)
            
            "Thread-33" prio=5 tid=0xf32b40 nid=0x4d in Object.wait() [e1881000..e18819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e8df1938> (a org.jboss.mx.util.RunnableScheduler)
             at org.jboss.mx.util.RunnableScheduler.waitOutstanding(RunnableScheduler.java:188)
             - locked <e8df1938> (a org.jboss.mx.util.RunnableScheduler)
             at org.jboss.mx.util.RunnableScheduler.run(RunnableScheduler.java:93)
             at java.lang.Thread.run(Thread.java:536)
            
            "Connection Consumer for dest QUEUE.configuration/configurationRequestQueue" prio=5 tid=0xf92038 nid=0x47 in Object.wait() [e2481000..e24819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e8d01060> (a java.util.LinkedList)
             at java.lang.Object.wait(Object.java:426)
             at org.jboss.mq.SpyConnectionConsumer.run(SpyConnectionConsumer.java:202)
             - locked <e8d01060> (a java.util.LinkedList)
             at java.lang.Thread.run(Thread.java:536)
            
            "Connection Consumer for dest QUEUE.diagnosis/dataCollectionQueue" prio=5 tid=0xc3e220 nid=0x45 in Object.wait() [e1d81000..e1d819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e8d01100> (a java.util.LinkedList)
             at java.lang.Object.wait(Object.java:426)
             at org.jboss.mq.SpyConnectionConsumer.run(SpyConnectionConsumer.java:202)
             - locked <e8d01100> (a java.util.LinkedList)
             at java.lang.Thread.run(Thread.java:536)
            
            "Connection Consumer for dest QUEUE.na/collectionResponseQueue" prio=5 tid=0x128dc8 nid=0x44 in Object.wait() [e1e81000..e1e819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e8d011a0> (a java.util.LinkedList)
             at java.lang.Object.wait(Object.java:426)
             at org.jboss.mq.SpyConnectionConsumer.run(SpyConnectionConsumer.java:202)
             - locked <e8d011a0> (a java.util.LinkedList)
             at java.lang.Thread.run(Thread.java:536)
            
            "Connection Consumer for dest QUEUE.na/collectionRequestQueue" prio=5 tid=0x82f4b8 nid=0x43 in Object.wait() [e1f81000..e1f819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e8d01240> (a java.util.LinkedList)
             at java.lang.Object.wait(Object.java:426)
             at org.jboss.mq.SpyConnectionConsumer.run(SpyConnectionConsumer.java:202)
             - locked <e8d01240> (a java.util.LinkedList)
             at java.lang.Thread.run(Thread.java:536)
            
            "Connection Consumer for dest QUEUE.na/storageRequestQueue" prio=5 tid=0x319db0 nid=0x42 in Object.wait() [e2081000..e20819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e8d012e0> (a java.util.LinkedList)
             at java.lang.Object.wait(Object.java:426)
             at org.jboss.mq.SpyConnectionConsumer.run(SpyConnectionConsumer.java:202)
             - locked <e8d012e0> (a java.util.LinkedList)
             at java.lang.Thread.run(Thread.java:536)
            
            "StandardManager[/web-console]" daemon prio=5 tid=0x29d108 nid=0x35 waiting on condition [e2d81000..e2d819bc]
             at java.lang.Thread.sleep(Native Method)
             at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
             at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
             at java.lang.Thread.run(Thread.java:536)
            
            "StandardManager[/jmx-console]" daemon prio=5 tid=0xbb6b10 nid=0x31 waiting on condition [e2581000..e25819bc]
             at java.lang.Thread.sleep(Native Method)
             at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
             at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
             at java.lang.Thread.run(Thread.java:536)
            
            "UILServerILService Accept Thread" prio=5 tid=0xbae330 nid=0x30 runnable [e2681000..e26819bc]
             at java.net.PlainSocketImpl.socketAccept(Native Method)
             at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
             - locked <e85a81e8> (a java.net.PlainSocketImpl)
             at java.net.ServerSocket.implAccept(ServerSocket.java:439)
             at java.net.ServerSocket.accept(ServerSocket.java:410)
             at org.jboss.mq.il.uil2.UILServerILService.run(UILServerILService.java:118)
             at java.lang.Thread.run(Thread.java:536)
            
            "RMI TCP Accept-0" daemon prio=5 tid=0xbb5678 nid=0x2f runnable [e2781000..e27819bc]
             at java.net.PlainSocketImpl.socketAccept(Native Method)
             at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
             - locked <e85a8320> (a java.net.PlainSocketImpl)
             at java.net.ServerSocket.implAccept(ServerSocket.java:439)
             at java.net.ServerSocket.accept(ServerSocket.java:410)
             at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
             at java.lang.Thread.run(Thread.java:536)
            
            "OIL2 Worker Server" prio=5 tid=0xba7d88 nid=0x2e runnable [e2881000..e28819bc]
             at java.net.PlainSocketImpl.socketAccept(Native Method)
             at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
             - locked <e85a8448> (a java.net.PlainSocketImpl)
             at java.net.ServerSocket.implAccept(ServerSocket.java:439)
             at java.net.ServerSocket.accept(ServerSocket.java:410)
             at org.jboss.mq.il.oil2.OIL2ServerILService.run(OIL2ServerILService.java:364)
             at java.lang.Thread.run(Thread.java:536)
            
            "Thread-17" prio=5 tid=0xb056a8 nid=0x2d in Object.wait() [e2981000..e29819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e85a8560> (a java.util.LinkedList)
             at java.lang.Object.wait(Object.java:426)
             at org.jboss.resource.connectionmanager.PoolFiller$1.run(PoolFiller.java:75)
             - locked <e85a8560> (a java.util.LinkedList)
             at java.lang.Thread.run(Thread.java:536)
            
            "IdleRemover" prio=5 tid=0xb26130 nid=0x2c in Object.wait() [e2a81000..e2a819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e85a85d0> (a java.util.ArrayList)
             at org.jboss.resource.connectionmanager.IdleRemover$1.run(IdleRemover.java:66)
             - locked <e85a85d0> (a java.util.ArrayList)
             at java.lang.Thread.run(Thread.java:536)
            
            "Thread-16" daemon prio=5 tid=0xb06db0 nid=0x2b in Object.wait() [e2b81000..e2b819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e85a8640> (a org.jboss.util.timeout.TimeoutFactory)
             at org.jboss.util.timeout.TimeoutFactory.doWork(TimeoutFactory.java:503)
             - locked <e85a8640> (a org.jboss.util.timeout.TimeoutFactory)
             at org.jboss.util.timeout.TimeoutFactory.access$300(TimeoutFactory.java:27)
             at org.jboss.util.timeout.TimeoutFactory$1.run(TimeoutFactory.java:542)
            
            "Thread-15" daemon prio=5 tid=0xb93998 nid=0x2a in Object.wait() [e2c81000..e2c819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e85a86a8> (a java.util.TaskQueue)
             at java.util.TimerThread.mainLoop(Timer.java:429)
             - locked <e85a86a8> (a java.util.TaskQueue)
             at java.util.TimerThread.run(Timer.java:382)
            
            "JBossMQ Cache Reference Softner" daemon prio=5 tid=0xb86940 nid=0x29 in Object.wait() [e3401000..e34019bc]
             at java.lang.Object.wait(Native Method)
             at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
             - locked <e82a9a50> (a java.lang.ref.ReferenceQueue$Lock)
             at org.jboss.mq.server.MessageCache.run(MessageCache.java:225)
             at java.lang.Thread.run(Thread.java:536)
            
            "StandardManager[/jbossmq-httpil]" daemon prio=5 tid=0x848c90 nid=0x25 waiting on condition [e3101000..e31019bc]
             at java.lang.Thread.sleep(Native Method)
             at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
             at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
             at java.lang.Thread.run(Thread.java:536)
            
            "StandardManager[/jboss-net]" daemon prio=5 tid=0x397c38 nid=0x24 waiting on condition [e3201000..e32019bc]
             at java.lang.Thread.sleep(Native Method)
             at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
             at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
             at java.lang.Thread.run(Thread.java:536)
            
            "StandardManager[/invoker]" daemon prio=5 tid=0xcb348 nid=0x23 waiting on condition [e4481000..e44819bc]
             at java.lang.Thread.sleep(Native Method)
             at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
             at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
             at java.lang.Thread.run(Thread.java:536)
            
            "TP-Monitor" daemon prio=5 tid=0x9c0728 nid=0x1f in Object.wait() [e3501000..e35019bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e7f7a248> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
             at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:560)
             - locked <e7f7a248> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
             at java.lang.Thread.run(Thread.java:536)
            
            "TP-Processor4" daemon prio=5 tid=0x9bfd90 nid=0x1e runnable [e3601000..e36019bc]
             at java.net.PlainSocketImpl.socketAccept(Native Method)
             at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
             - locked <e7f78438> (a java.net.PlainSocketImpl)
             at java.net.ServerSocket.implAccept(ServerSocket.java:439)
             at java.net.ServerSocket.accept(ServerSocket.java:410)
             at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:287)
             at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:581)
             at org.apache.jk.common.SocketAcceptor.runIt(ChannelSocket.java:767)
             at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
             at java.lang.Thread.run(Thread.java:536)
            
            "TP-Processor3" daemon prio=5 tid=0x9c20d8 nid=0x1d in Object.wait() [e3701000..e37019bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e7f79ec0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Object.wait(Object.java:426)
             at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
             - locked <e7f79ec0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Thread.run(Thread.java:536)
            
            "TP-Processor2" daemon prio=5 tid=0x9a4458 nid=0x1c in Object.wait() [e3801000..e38019bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e7f79b68> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Object.wait(Object.java:426)
             at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
             - locked <e7f79b68> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Thread.run(Thread.java:536)
            
            "TP-Processor1" daemon prio=5 tid=0x9a42c0 nid=0x1b in Object.wait() [e3901000..e39019bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e7f799b0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Object.wait(Object.java:426)
             at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
             - locked <e7f799b0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Thread.run(Thread.java:536)
            
            "TP-Monitor" daemon prio=5 tid=0x9bf3f8 nid=0x1a in Object.wait() [e3a01000..e3a019bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e7f6abc8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
             at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:560)
             - locked <e7f6abc8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
             at java.lang.Thread.run(Thread.java:536)
            
            "TP-Processor4" daemon prio=5 tid=0x9a5348 nid=0x19 in Object.wait() [e3b01000..e3b019bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e7f6a6a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Object.wait(Object.java:426)
             at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
             - locked <e7f6a6a0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Thread.run(Thread.java:536)
            
            "TP-Processor3" daemon prio=5 tid=0x9a51b0 nid=0x18 in Object.wait() [e3c01000..e3c019bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e7f6a4e8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Object.wait(Object.java:426)
             at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
             - locked <e7f6a4e8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Thread.run(Thread.java:536)
            
            "TP-Processor2" daemon prio=5 tid=0x9a4818 nid=0x17 in Object.wait() [e3d01000..e3d019bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e7f6a330> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Object.wait(Object.java:426)
             at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
             - locked <e7f6a330> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
             at java.lang.Thread.run(Thread.java:536)
            
            "TP-Processor1" daemon prio=5 tid=0x981188 nid=0x16 runnable [e3e01000..e3e019bc]
             at java.net.SocketInputStream.socketRead0(Native Method)
             at java.net.SocketInputStream.read(SocketInputStream.java:129)
             at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:767)
             at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:428)
             at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:743)
             at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:549)
             at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:605)
             at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
             at java.lang.Thread.run(Thread.java:536)
            
            "Thread-4" prio=5 tid=0x716c60 nid=0x15 runnable [e4281000..e42819bc]
             at java.net.PlainSocketImpl.socketAccept(Native Method)
             at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
             - locked <e7fed5e8> (a java.net.PlainSocketImpl)
             at java.net.ServerSocket.implAccept(ServerSocket.java:439)
             at java.net.ServerSocket.accept(ServerSocket.java:410)
             at org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:211)
             at java.lang.Thread.run(Thread.java:536)
            
            "RMI TCP Accept-4444" daemon prio=5 tid=0x779db0 nid=0x14 runnable [e4381000..e43819bc]
             at java.net.PlainSocketImpl.socketAccept(Native Method)
             at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
             - locked <e7fed710> (a java.net.PlainSocketImpl)
             at java.net.ServerSocket.implAccept(ServerSocket.java:439)
             at java.net.ServerSocket.accept(ServerSocket.java:410)
             at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
             at java.lang.Thread.run(Thread.java:536)
            
            "GC Daemon" daemon prio=2 tid=0x772200 nid=0x12 in Object.wait() [e4581000..e45819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e7d7c650> (a sun.misc.GC$LatencyLock)
             at sun.misc.GC$Daemon.run(GC.java:100)
             - locked <e7d7c650> (a sun.misc.GC$LatencyLock)
            
            "RMI Reaper" prio=5 tid=0x7717c0 nid=0x11 in Object.wait() [e4681000..e46819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e7d7bb38> (a java.lang.ref.ReferenceQueue$Lock)
             at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
             - locked <e7d7bb38> (a java.lang.ref.ReferenceQueue$Lock)
             at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
             at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330)
             at java.lang.Thread.run(Thread.java:536)
            
            "Thread-3" daemon prio=5 tid=0x76fae8 nid=0x10 in Object.wait() [e4781000..e47819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e7d7be08> (a java.util.TaskQueue)
             at java.lang.Object.wait(Object.java:426)
             at java.util.TimerThread.mainLoop(Timer.java:403)
             - locked <e7d7be08> (a java.util.TaskQueue)
             at java.util.TimerThread.run(Timer.java:382)
            
            "RMI TCP Accept-1098" daemon prio=5 tid=0x76f950 nid=0xf runnable [e4881000..e48819bc]
             at java.net.PlainSocketImpl.socketAccept(Native Method)
             at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
             - locked <e7d7bb98> (a java.net.PlainSocketImpl)
             at java.net.ServerSocket.implAccept(ServerSocket.java:439)
             at java.net.ServerSocket.accept(ServerSocket.java:410)
             at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
             at java.lang.Thread.run(Thread.java:536)
            
            "Thread-2" daemon prio=5 tid=0x7701d8 nid=0xe runnable [e4981000..e49819bc]
             at java.net.PlainSocketImpl.socketAccept(Native Method)
             at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
             - locked <e7d77d78> (a java.net.PlainSocketImpl)
             at java.net.ServerSocket.implAccept(ServerSocket.java:439)
             at java.net.ServerSocket.accept(ServerSocket.java:410)
             at org.jboss.web.WebServer.run(WebServer.java:258)
             at org.jboss.web.ThreadPool$Worker.run(ThreadPool.java:163)
            
            "ScannerThread" daemon prio=5 tid=0x76ee28 nid=0xd waiting on condition [e4a81000..e4a819bc]
             at java.lang.Thread.sleep(Native Method)
             at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:218)
             at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:191)
            
            "Thread-1" daemon prio=5 tid=0x626de0 nid=0xc in Object.wait() [e4b81000..e4b819bc]
             at java.lang.Object.wait(Native Method)
             - waiting on <e7d7ca48> (a java.util.TaskQueue)
             at java.util.TimerThread.mainLoop(Timer.java:429)
             - locked <e7d7ca48> (a java.util.TaskQueue)
             at java.util.TimerThread.run(Timer.java:382)
            
            "DestroyJavaVM" prio=5 tid=0x2c928 nid=0x1 waiting on condition [0..ffbeedbc]
            
            "Signal Dispatcher" daemon prio=10 tid=0xb8c38 nid=0x9 waiting on condition [0..0]
            
            "Finalizer" daemon prio=8 tid=0xb5278 nid=0x6 in Object.wait() [fa281000..fa2819bc]
             at java.lang.Object.wait(Native Method)
             at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
             - locked <e7d7d5e0> (a java.lang.ref.ReferenceQueue$Lock)
             at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
             at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
            
            "Reference Handler" daemon prio=10 tid=0xb3920 nid=0x5 in Object.wait() [fa381000..fa3819bc]
             at java.lang.Object.wait(Native Method)
             at java.lang.Object.wait(Object.java:426)
             at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:113)
             - locked <e7d7cbd0> (a java.lang.ref.Reference$Lock)
            
            "VM Thread" prio=5 tid=0xb2488 nid=0x4 runnable
            
            "VM Periodic Task Thread" prio=10 tid=0xb7990 nid=0x7 waiting on condition
            "Suspend Checker Thread" prio=10 tid=0xb82e0 nid=0x8 runnable
            


            For the moment, i don't have log files with JMS tracing enabled.

            I guess the post you were refering to is this one http://www.jboss.org/index.html?module=bb&op=viewtopic&t=48112.
            I will try to play around with the solaris settings.

            By the way, we already switched to java:/ConnectionFactory. But i would still like to know why the TCP invokers don't work ;-)




            • 3. Re: deadlock in invocation layers on solaris
              Adrian Brock Master

              The UIL2 trace shows you cant even make a proper connection.

              "Thread-34" daemon prio=5 tid=0xf88ed0 nid=0x4e in Object.wait() [e1b80000..e1b819bc]
               at java.lang.Object.wait(Native Method)
               at java.lang.Object.wait(Object.java:426)
               at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:240)
               - locked <e91ef420> (a org.jboss.mq.il.uil2.msgs.CheckUserMsg)
               at org.jboss.mq.il.uil2.SocketManager.sendMessage(SocketManager.java:189)
               at org.jboss.mq.il.uil2.UILServerIL.authenticate(UILServerIL.java:302)
              


              This is the first operation on the network.

              The fact that you have a write task and no read task, suggests it crashed on the
              server while trying to do the handshake:
              "UIL2.SocketManager.WriteTask#3928"

              I would expect an error or warning in the log telling you the error that caused
              the read task to crash.

              NOTE: This has been fixed in 3.2.4, if the read task crashes during the initial handshake,
              it now stops the write task.