Hello,

I am having problems with a very simple JMS application.

I have a message producer, which sends ObjectMessages in a
while(true) -loop and a multi-threaded consumer
application, which receives messages from the queue and
makes some processing for each received message.

After about 10k messages the whole system just locks up.
When I take a thread dump, it seems that all components
(i.e. JBoss, the producer and the consumer) are locked
because of pinger. Thread dumps can be found from the end
of this message.

Consumer is waiting for a lock acquired by the Connection
Monitor Thread, and the same applies to the producer.

My application is designed according to the copy-paste
design pattern ;) The original source is Sun's JMS
tutorial, I just added some multithreading to the consumer
side.

Is there something wrong with my code or with JBoss?

Best regards, Matti

Thread dumps:

The Consumer:

"PooledThread-7" daemon prio=1 tid=0x0x81a5008 nid=0x6db4 waiting for monitor entry [bddff000..bddff8ac]
at org.jboss.mq.il.oil.OILServerIL.acknowledge(OILServerIL.java:177)
- waiting to lock <0x44659470> (a org.jboss.mq.il.oil.OILServerIL)
at org.jboss.mq.Connection.send(Connection.java:786)
at org.jboss.mq.SpyMessage.doAcknowledge(SpyMessage.java:537)
at org.jboss.mq.SpyMessageConsumer.preProcessMessage(SpyMessageConsumer.java:653)
at org.jboss.mq.SpyMessageConsumer.getMessage(SpyMessageConsumer.java:614)
- locked <0x44659530> (a java.util.LinkedList)
at org.jboss.mq.SpyMessageConsumer.receive(SpyMessageConsumer.java:257)
- locked <0x44659530> (a java.util.LinkedList)
at fi.mermit.gateway.jms.JMSBroker.execute(JMSBroker.java:332)
at fi.mermit.tools.thread.SimplePool$WorkerThread.run(SimplePool.java:183)

"Connection Monitor Thread" daemon prio=1 tid=0x0x8225370 nid=0x6dc3 runnable [bc7ff000..bc7ff8ac]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:116)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
- locked <0x446593d8> (a java.io.BufferedInputStream)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2118)
at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2301)
at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2368)
at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2440)
at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2589)
at java.io.ObjectInputStream.readByte(ObjectInputStream.java:837)
at org.jboss.mq.il.oil.OILServerIL.waitAnswer(OILServerIL.java:521)
at org.jboss.mq.il.oil.OILServerIL.ping(OILServerIL.java:395)
- locked <0x44659470> (a org.jboss.mq.il.oil.OILServerIL)
at org.jboss.mq.Connection.pingServer(Connection.java:1021)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1183)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:536)


The Producer:

"main" prio=1 tid=0x0x8051a98 nid=0x6dc4 waiting for monitor entry [bfffc000..bfffd118]
at org.jboss.mq.il.oil.OILServerIL.addMessage(OILServerIL.java:193)
- waiting to lock <0x446423d8> (a org.jboss.mq.il.oil.OILServerIL)
at org.jboss.mq.Connection.sendToServer(Connection.java:1119)
at org.jboss.mq.SpySession.sendMessage(SpySession.java:444)
at org.jboss.mq.SpyQueueSender.internalSend(SpyQueueSender.java:118)
at org.jboss.mq.SpyQueueSender.send(SpyQueueSender.java:68)
at fi.mermit.gateway.diagnostic.JMSConsumer.flood(JMSConsumer.java:122)
at fi.mermit.gateway.diagnostic.JMSConsumer.main(JMSConsumer.java:79)

"Connection Monitor Thread" daemon prio=1 tid=0x0x81f3cc8 nid=0x6dd5 runnable [be1ff000..be1ff8ac]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:116)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
- locked <0x446459a8> (a java.io.BufferedInputStream)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2118)
at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2301)
at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2368)
at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2440)
at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2589)
at java.io.ObjectInputStream.readByte(ObjectInputStream.java:837)
at org.jboss.mq.il.oil.OILServerIL.waitAnswer(OILServerIL.java:521)
at org.jboss.mq.il.oil.OILServerIL.ping(OILServerIL.java:395)
- locked <0x446423d8> (a org.jboss.mq.il.oil.OILServerIL)
at org.jboss.mq.Connection.pingServer(Connection.java:1021)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1183)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:536)


JBoss:

All Message Pusher threads are waiting for a lock, as follows:

"Message Pushers-6" daemon prio=1 tid=0x0x8634a48 nid=0x6ddb waiting for monitor entry [ba3ff000..ba3ff8ac]
at org.jboss.mq.il.oil.OILClientIL.receive(OILClientIL.java:127)
- waiting to lock <0x44da9790> (a org.jboss.mq.il.oil.OILClientIL)
at org.jboss.mq.server.ClientConsumer.doWork(ClientConsumer.java:277)
at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java:230)

...except one thread:

"Message Pushers-7" daemon prio=1 tid=0x0x8635688 nid=0x6ddc runnable [ba1ff000..ba1ff8ac]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:116)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
- locked <0x44daa078> (a java.io.BufferedInputStream)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2118)
at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2301)
at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2368)
at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2440)
at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2589)
at java.io.ObjectInputStream.readByte(ObjectInputStream.java:837)
at org.jboss.mq.il.oil.OILClientIL.waitAnswer(OILClientIL.java:200)
at org.jboss.mq.il.oil.OILClientIL.receive(OILClientIL.java:141)
- locked <0x44da9790> (a org.jboss.mq.il.oil.OILClientIL)
at org.jboss.mq.server.ClientConsumer.doWork(ClientConsumer.java:277)
at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java:230)

And the trace for the Pinger Thread looks as follows:

"OIL Worker-1" prio=1 tid=0x0x8227698 nid=0x6dc1 waiting for monitor entry [bb3ff000..bb3ff8ac]
at org.jboss.mq.il.oil.OILClientIL.pong(OILClientIL.java:112)
- waiting to lock <0x44da9790> (a org.jboss.mq.il.oil.OILClientIL)
at org.jboss.mq.server.JMSDestinationManager.ping(JMSDestinationManager.java:853)
- locked <0x44c70d00> (a org.jboss.mq.server.JMSDestinationManager)
at org.jboss.mq.server.JMSServerInterceptorSupport.ping(JMSServerInterceptorSupport.java:308)
at org.jboss.mq.server.TracingInterceptor.ping(TracingInterceptor.java:630)
at org.jboss.mq.server.JMSServerInvoker.ping(JMSServerInvoker.java:310)
at org.jboss.mq.il.oil.OILServerILService$Client.run(OILServerILService.java:309)
at java.lang.Thread.run(Thread.java:536)