deadlock in invocation layers on solaris
Hello all,
We are experiencing some annoying deadlock behavior when trying to send messages a queue. The code we use for sending a message is pretty straightforward:
public static void sendRequestsOnQueue(QueueConnectionFactory factory, Queue queue, List requests, int deliveryMode) throws JMSException {
QueueConnection queueConnection = factory.createQueueConnection();
try {
QueueSession qs = queueConnection.createQueueSession(false, javax.jms.Session.AUTO_ACKNOWLEDGE);
try {
QueueSender sender = qs.createSender(queue);
try {
sender.setDeliveryMode(deliveryMode);
for (Iterator iterator = requests.iterator(); iterator.hasNext();) {
Serializable request = (Serializable) iterator.next();
sender.send(qs.createObjectMessage(request));
}
m_log.debug("Posted " + requests.size() + " jobs on queue " + queue.getQueueName());
} finally {
sender.close();
}
} finally {
qs.close();
}
} finally {
queueConnection.close();
}
}
Deadlocks appear with both OIL and UIL2 invokers. I will add the threaddump for the OIL case below. Only the most important threads are show
"OILClientILService-164" daemon prio=5 tid=0x814b48 nid=0x4ec runnable [e0b81000..e0b819bc] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:183) at java.io.BufferedInputStream.read(BufferedInputStream.java:201) - locked <e9334180> (a java.io.BufferedInputStream) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2133) at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2316) at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2383) at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2455) at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2604) at java.io.ObjectInputStream.readByte(ObjectInputStream.java:845) at org.jboss.mq.il.oil.OILClientILService.run(OILClientILService.java:183) at java.lang.Thread.run(Thread.java:536) "OIL Worker-164" prio=5 tid=0xb1f828 nid=0x4e9 runnable [e1580000..e15819bc] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:183) at java.io.BufferedInputStream.read(BufferedInputStream.java:201) - locked <e9332618> (a java.io.BufferedInputStream) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2133) at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2316) at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2383) at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2455) at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2604) at java.io.ObjectInputStream.readByte(ObjectInputStream.java:845) at org.jboss.mq.il.oil.OILClientIL.waitAnswer(OILClientIL.java:200) at org.jboss.mq.il.oil.OILClientIL.close(OILClientIL.java:74) - locked <e9331790> (a org.jboss.mq.il.oil.OILClientIL) at org.jboss.mq.server.JMSDestinationManager.connectionClosing(JMSDestinationManager.java:571) at org.jboss.mq.server.JMSServerInterceptorSupport.connectionClosing(JMSServerInterceptorSupport.java:113) at org.jboss.mq.security.ServerSecurityInterceptor.connectionClosing(ServerSecurityInterceptor.java:61) at org.jboss.mq.server.TracingInterceptor.connectionClosing(TracingInterceptor.java:185) at org.jboss.mq.server.JMSServerInvoker.connectionClosing(JMSServerInvoker.java:113) at org.jboss.mq.il.oil.OILServerILService$Client.run(OILServerILService.java:264) at java.lang.Thread.run(Thread.java:536) "Message Pushers-1" daemon prio=5 tid=0x10c0140 nid=0x54 in Object.wait() [e1281000..e12819bc] at java.lang.Object.wait(Native Method) - waiting on <e8e13ec0> (a java.util.LinkedList) at java.lang.Object.wait(Object.java:426) at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java:205) at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java:224) - locked <e8e13ec0> (a java.util.LinkedList) "Connection Monitor Thread" daemon prio=5 tid=0x10beeb0 nid=0x52 waiting for monitor entry [e1481000..e14819bc] at org.jboss.mq.il.oil.OILServerIL.ping(OILServerIL.java:421) - waiting to lock <e932d978> (a org.jboss.mq.il.oil.OILServerIL) at org.jboss.mq.Connection.pingServer(Connection.java:1141) at org.jboss.mq.Connection$PingTask.run(Connection.java:1307) at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364) at java.lang.Thread.run(Thread.java:536) "Thread-33" daemon prio=5 tid=0x10b42a0 nid=0x4e in Object.wait() [e1781000..e17819bc] at java.lang.Object.wait(Native Method) - waiting on <e8e62f20> (a org.jboss.mx.util.ThreadPool$Worker) at java.lang.Object.wait(Object.java:426) at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:243) - locked <e8e62f20> (a org.jboss.mx.util.ThreadPool$Worker) "Thread-32" daemon prio=5 tid=0xeca348 nid=0x4d runnable [e1880000..e18819bc] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:183) at java.io.BufferedInputStream.read(BufferedInputStream.java:201) - locked <e932f518> (a java.io.BufferedInputStream) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2133) at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2316) at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2383) at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2455) at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2604) at java.io.ObjectInputStream.readByte(ObjectInputStream.java:845) at org.jboss.mq.il.oil.OILServerIL.waitAnswer(OILServerIL.java:601) at org.jboss.mq.il.oil.OILServerIL.connectionClosing(OILServerIL.java:334) - locked <e932d978> (a org.jboss.mq.il.oil.OILServerIL) at org.jboss.mq.Connection.close(Connection.java:491) - locked <e932d738> (a org.jboss.mq.SpyConnection) at com.alcatel.util.jms.JmsUtil.sendRequestsOnQueue(JmsUtil.java:63) at com.alcatel.ni.datacollection.collection.SchedulableCollector.sendAsamCollectionRequest(SchedulableCollector.java:292) at com.alcatel.ni.datacollection.collection.SchedulableCollector.collect(SchedulableCollector.java:218) at sun.reflect.GeneratedMethodAccessor204.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546) at org.jboss.varia.scheduler.Scheduler$MBeanListener.handleNotification(Scheduler.java:1354) at org.jboss.mx.server.NotificationListenerProxy.handleNotification(NotificationListenerProxy.java:69) at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:95) at javax.management.timer.Timer.sendNotifications(Timer.java:441) at javax.management.timer.Timer.access$000(Timer.java:31) at javax.management.timer.Timer$RegisteredNotification.doRun(Timer.java:612) at org.jboss.mx.util.SchedulableRunnable.run(SchedulableRunnable.java:164) at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:225)
Currently, we worked around the problem by using the JVM invoker and skipping to complete socket layer. Although it fits our needs currently, it will give problems the day we need to be distributed.
We are using jboss 3.2.3 on solaris 2.8 running on a dual process E250. Another fact noting is that we were not able to reproduce this behavior on an NT box.