1 Reply Latest reply on Aug 22, 2002 5:54 PM by Andreas Schaefer

    Pinger hangs up the system

    Matti Kokkola Newbie

      Hello,

      I am having problems with a very simple JMS application.

      I have a message producer, which sends ObjectMessages in a
      while(true) -loop and a multi-threaded consumer
      application, which receives messages from the queue and
      makes some processing for each received message.

      After about 10k messages the whole system just locks up.
      When I take a thread dump, it seems that all components
      (i.e. JBoss, the producer and the consumer) are locked
      because of pinger. Thread dumps can be found from the end
      of this message.

      Consumer is waiting for a lock acquired by the Connection
      Monitor Thread, and the same applies to the producer.

      My application is designed according to the copy-paste
      design pattern ;) The original source is Sun's JMS
      tutorial, I just added some multithreading to the consumer
      side.

      Is there something wrong with my code or with JBoss?

      Best regards, Matti

      Thread dumps:

      The Consumer:

      "PooledThread-7" daemon prio=1 tid=0x0x81a5008 nid=0x6db4 waiting for monitor entry [bddff000..bddff8ac]
      at org.jboss.mq.il.oil.OILServerIL.acknowledge(OILServerIL.java:177)
      - waiting to lock <0x44659470> (a org.jboss.mq.il.oil.OILServerIL)
      at org.jboss.mq.Connection.send(Connection.java:786)
      at org.jboss.mq.SpyMessage.doAcknowledge(SpyMessage.java:537)
      at org.jboss.mq.SpyMessageConsumer.preProcessMessage(SpyMessageConsumer.java:653)
      at org.jboss.mq.SpyMessageConsumer.getMessage(SpyMessageConsumer.java:614)
      - locked <0x44659530> (a java.util.LinkedList)
      at org.jboss.mq.SpyMessageConsumer.receive(SpyMessageConsumer.java:257)
      - locked <0x44659530> (a java.util.LinkedList)
      at fi.mermit.gateway.jms.JMSBroker.execute(JMSBroker.java:332)
      at fi.mermit.tools.thread.SimplePool$WorkerThread.run(SimplePool.java:183)

      "Connection Monitor Thread" daemon prio=1 tid=0x0x8225370 nid=0x6dc3 runnable [bc7ff000..bc7ff8ac]
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:116)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
      - locked <0x446593d8> (a java.io.BufferedInputStream)
      at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2118)
      at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2301)
      at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2368)
      at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2440)
      at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2589)
      at java.io.ObjectInputStream.readByte(ObjectInputStream.java:837)
      at org.jboss.mq.il.oil.OILServerIL.waitAnswer(OILServerIL.java:521)
      at org.jboss.mq.il.oil.OILServerIL.ping(OILServerIL.java:395)
      - locked <0x44659470> (a org.jboss.mq.il.oil.OILServerIL)
      at org.jboss.mq.Connection.pingServer(Connection.java:1021)
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1183)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
      at java.lang.Thread.run(Thread.java:536)


      The Producer:

      "main" prio=1 tid=0x0x8051a98 nid=0x6dc4 waiting for monitor entry [bfffc000..bfffd118]
      at org.jboss.mq.il.oil.OILServerIL.addMessage(OILServerIL.java:193)
      - waiting to lock <0x446423d8> (a org.jboss.mq.il.oil.OILServerIL)
      at org.jboss.mq.Connection.sendToServer(Connection.java:1119)
      at org.jboss.mq.SpySession.sendMessage(SpySession.java:444)
      at org.jboss.mq.SpyQueueSender.internalSend(SpyQueueSender.java:118)
      at org.jboss.mq.SpyQueueSender.send(SpyQueueSender.java:68)
      at fi.mermit.gateway.diagnostic.JMSConsumer.flood(JMSConsumer.java:122)
      at fi.mermit.gateway.diagnostic.JMSConsumer.main(JMSConsumer.java:79)

      "Connection Monitor Thread" daemon prio=1 tid=0x0x81f3cc8 nid=0x6dd5 runnable [be1ff000..be1ff8ac]
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:116)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
      - locked <0x446459a8> (a java.io.BufferedInputStream)
      at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2118)
      at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2301)
      at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2368)
      at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2440)
      at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2589)
      at java.io.ObjectInputStream.readByte(ObjectInputStream.java:837)
      at org.jboss.mq.il.oil.OILServerIL.waitAnswer(OILServerIL.java:521)
      at org.jboss.mq.il.oil.OILServerIL.ping(OILServerIL.java:395)
      - locked <0x446423d8> (a org.jboss.mq.il.oil.OILServerIL)
      at org.jboss.mq.Connection.pingServer(Connection.java:1021)
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1183)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
      at java.lang.Thread.run(Thread.java:536)


      JBoss:

      All Message Pusher threads are waiting for a lock, as follows:

      "Message Pushers-6" daemon prio=1 tid=0x0x8634a48 nid=0x6ddb waiting for monitor entry [ba3ff000..ba3ff8ac]
      at org.jboss.mq.il.oil.OILClientIL.receive(OILClientIL.java:127)
      - waiting to lock <0x44da9790> (a org.jboss.mq.il.oil.OILClientIL)
      at org.jboss.mq.server.ClientConsumer.doWork(ClientConsumer.java:277)
      at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java:230)

      ...except one thread:

      "Message Pushers-7" daemon prio=1 tid=0x0x8635688 nid=0x6ddc runnable [ba1ff000..ba1ff8ac]
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:116)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
      - locked <0x44daa078> (a java.io.BufferedInputStream)
      at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2118)
      at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2301)
      at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2368)
      at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2440)
      at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2589)
      at java.io.ObjectInputStream.readByte(ObjectInputStream.java:837)
      at org.jboss.mq.il.oil.OILClientIL.waitAnswer(OILClientIL.java:200)
      at org.jboss.mq.il.oil.OILClientIL.receive(OILClientIL.java:141)
      - locked <0x44da9790> (a org.jboss.mq.il.oil.OILClientIL)
      at org.jboss.mq.server.ClientConsumer.doWork(ClientConsumer.java:277)
      at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java:230)

      And the trace for the Pinger Thread looks as follows:

      "OIL Worker-1" prio=1 tid=0x0x8227698 nid=0x6dc1 waiting for monitor entry [bb3ff000..bb3ff8ac]
      at org.jboss.mq.il.oil.OILClientIL.pong(OILClientIL.java:112)
      - waiting to lock <0x44da9790> (a org.jboss.mq.il.oil.OILClientIL)
      at org.jboss.mq.server.JMSDestinationManager.ping(JMSDestinationManager.java:853)
      - locked <0x44c70d00> (a org.jboss.mq.server.JMSDestinationManager)
      at org.jboss.mq.server.JMSServerInterceptorSupport.ping(JMSServerInterceptorSupport.java:308)
      at org.jboss.mq.server.TracingInterceptor.ping(TracingInterceptor.java:630)
      at org.jboss.mq.server.JMSServerInvoker.ping(JMSServerInvoker.java:310)
      at org.jboss.mq.il.oil.OILServerILService$Client.run(OILServerILService.java:309)
      at java.lang.Thread.run(Thread.java:536)