Hey - Adrian bug [ 745527 ] race condition not fixed

[jamoville]

This bug is not fixed - I looked at the latest code in cvs and the code has not been changed. While you did fix a deadlock situation in the BasicQueue class you missed fixing the one I was mentioning. Specifically the methods:
hasUnackedMessages
and
checkRemovedSubscribers

when these methods are called by two different threads a deadlock happens. You should be synchronizing on the methods. You are causing too much context switching and it's killing performance. You also wouldn't have to be doing all those nested synchronizations.

here is the stack trace:
FOUND A JAVA LEVEL DEADLOCK:

----------------------------

"Thread-186":

waiting to lock monitor 0x8ed8f4 (object 0x3395af8, a java.util.HashSet),

which is locked by "Thread-184"

"Thread-184":

waiting to lock monitor 0x8eda14 (object 0x3395a68, a java.util.HashMap),

which is locked by "Thread-186"



JAVA STACK INFORMATION FOR THREADS LISTED ABOVE:

------------------------------------------------

Java Stack for "Thread-186":

==========

at org.jboss.mq.server.BasicQueue.checkRemovedSubscribers(BasicQueue.java:508)

at org.jboss.mq.server.BasicQueue.acknowledge(BasicQueue.java:404)

at org.jboss.mq.server.JMSQueue.acknowledge(JMSQueue.java:114)

at org.jboss.mq.server.ClientConsumer.acknowledge(ClientConsumer.java:328)

at org.jboss.mq.server.JMSDestinationManager.acknowledge(JMSDestinationManager.java:529)

at org.jboss.mq.server.JMSDestinationManager.acknowledge(JMSDestinationManager.java:513)

at org.jboss.mq.server.JMSServerInterceptorSupport.acknowledge(JMSServerInterceptorSupport.java:197)

at org.jboss.mq.server.TracingInterceptor.acknowledge(TracingInterceptor.java:405)

at org.jboss.mq.server.JMSServerInvoker.acknowledge(JMSServerInvoker.java:199)

at org.jboss.mq.il.uil2.ServerSocketManagerHandler.handleMsg(ServerSocketManagerHandler.java:81)

at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleMsg(SocketManager.java:355)

at org.jboss.mq.il.uil2.msgs.BaseMsg.run(BaseMsg.java:377)

at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:732)

at java.lang.Thread.run(Thread.java:479)

Java Stack for "Thread-184":

==========

at org.jboss.mq.server.BasicQueue.hasUnackedMessages(BasicQueue.java:500)

at org.jboss.mq.server.BasicQueue.removeSubscriber(BasicQueue.java:178)

at org.jboss.mq.server.JMSQueue.removeSubscriber(JMSQueue.java:60)

at org.jboss.mq.server.ClientConsumer.removeSubscription(ClientConsumer.java:253)

at org.jboss.mq.server.JMSDestinationManager.unsubscribe(JMSDestinationManager.java:624)

at org.jboss.mq.server.JMSServerInterceptorSupport.unsubscribe(JMSServerInterceptorSupport.java:249)

at org.jboss.mq.server.TracingInterceptor.unsubscribe(TracingInterceptor.java:548)

at org.jboss.mq.server.JMSServerInvoker.unsubscribe(JMSServerInvoker.java:251)

at org.jboss.mq.il.uil2.ServerSocketManagerHandler.handleMsg(ServerSocketManagerHandler.java:171)

at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleMsg(SocketManager.java:355)

at org.jboss.mq.il.uil2.msgs.BaseMsg.run(BaseMsg.java:377)

at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:732)

at java.lang.Thread.run(Thread.java:479)



Found 1 deadlock.

[jamoville]

Just an FYI - each synch causes four byte code instruction and you have synch code that has four syncs. this will generate 16 byte code instruction. Sychronizing on the method will reduce this by 75%

[monu]

Hi all,

I am also getting the same error. We have JBoss running under HP-UX 11.11. When we try to send message in high speed to the queue and receive the message in same speed. After 1-2 hours all the receiver threads hangs and in jboss console , it displays the error:

"CPMessageDistributor3" prio=10 tid=0x00575ae0 nid=1107 lwp_id=4869387 waiting for monitor entry [0x4d713000..0x4d713530]
at org.jboss.mq.server.BasicQueue.receive(BasicQueue.java:238)
- waiting to lock <0x591835f8> (a java.util.HashSet)
at org.jboss.mq.server.JMSQueue.receive(JMSQueue.java:130)
at org.jboss.mq.server.ClientConsumer.receive(ClientConsumer.java:225)
at org.jboss.mq.server.JMSDestinationManager.receive(JMSDestinationManager.java:672)
at org.jboss.mq.server.JMSServerInterceptorSupport.receive(JMSServerInterceptorSupport.java:225)
at org.jboss.mq.security.ServerSecurityInterceptor.receive(ServerSecurityInterceptor.java:103)
at org.jboss.mq.server.TracingInterceptor.receive(TracingInterceptor.java:478)
at org.jboss.mq.server.JMSServerInvoker.receive(JMSServerInvoker.java:227)
at org.jboss.mq.il.jvm.JVMServerIL.receive(JVMServerIL.java:245)
at org.jboss.mq.Connection.receive(Connection.java:1166)
at org.jboss.mq.SpyMessageConsumer.receive(SpyMessageConsumer.java:266)
- locked <0x6001e0f0> (a java.util.LinkedList)
at com.mobilgw.sam.executionengine.queue.CPMessageDistributor.run(CPMessageDistributor.java:153)
at java.lang.Thread.run(Thread.java:536)

Pleaae give your valuable suggestions.

Thanks in advance.

Monu

[adrian.brock]

I simplified the synchronization for jboss-3.2.2RC1
here is the diff
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/jboss/jbossmq/src/main/org/jboss/mq/server/BasicQueue.java.diff?r1=1.20.2.10&r2=1.20.2.11

The comments about bytecode instructions are
illconceived.

Replacing a few bytecodes with a global synchronization
will only slow things down as everything waits on a
single point of contention.

Regards,
Adrian

[monu]

Adrian, Thank you very much.

I am using Jboss3.2.1 and facing the reported problem. I think Jboss3.2.2.RCx is not ready for production. How can I use the updated BasicQueue class with JBoss3.2.1.
Please give your valuable suggestions. I am facing very big problem because of deadlock issue.

Thanks,
Monu

[adrian.brock]

You can get the 3.2.1 source and apply the diff
I pointed to.

Regards,
Adrian

[monu]

Adrin, Thanx so much.
I tried to put chnages in BasicQueue.java of Jboss3.2.1 source but it seems there is a big difference in between jboss3.2.2 RC2 and jboss3.2.1 BasicQueue.java, that is why i clould not merge these properly. Could you please send me the BasicQueue.java in which you have fixed the deadlock bug.

Thanks,
Monu

[adrian.brock]

The version where I fixed is the link I provided.

There were other changes around the same time
for scheduled delivery that probably make it difficult
to apply.

Why don't you download jbossmq-3.2.2RC4
take jbossmq.jar and jbossmq-client.jar from it.
As far as I'm concerned that is stable (even if the
rest of jboss in the release candidate still has issues).

NOTE: If you use jbossall-client.jar you'll need
to update the org/jboss/mq classes.

Regards,
Adrian