7 Replies Latest reply on Sep 25, 2003 7:49 AM by Adrian Brock

    Hey - Adrian bug [ 745527 ] race condition not fixed

    john Newbie

      This bug is not fixed - I looked at the latest code in cvs and the code has not been changed. While you did fix a deadlock situation in the BasicQueue class you missed fixing the one I was mentioning. Specifically the methods:
      hasUnackedMessages
      and
      checkRemovedSubscribers

      when these methods are called by two different threads a deadlock happens. You should be synchronizing on the methods. You are causing too much context switching and it's killing performance. You also wouldn't have to be doing all those nested synchronizations.

      here is the stack trace:
      FOUND A JAVA LEVEL DEADLOCK:

      ----------------------------

      "Thread-186":

      waiting to lock monitor 0x8ed8f4 (object 0x3395af8, a java.util.HashSet),

      which is locked by "Thread-184"

      "Thread-184":

      waiting to lock monitor 0x8eda14 (object 0x3395a68, a java.util.HashMap),

      which is locked by "Thread-186"



      JAVA STACK INFORMATION FOR THREADS LISTED ABOVE:

      ------------------------------------------------

      Java Stack for "Thread-186":

      ==========

      at org.jboss.mq.server.BasicQueue.checkRemovedSubscribers(BasicQueue.java:508)

      at org.jboss.mq.server.BasicQueue.acknowledge(BasicQueue.java:404)

      at org.jboss.mq.server.JMSQueue.acknowledge(JMSQueue.java:114)

      at org.jboss.mq.server.ClientConsumer.acknowledge(ClientConsumer.java:328)

      at org.jboss.mq.server.JMSDestinationManager.acknowledge(JMSDestinationManager.java:529)

      at org.jboss.mq.server.JMSDestinationManager.acknowledge(JMSDestinationManager.java:513)

      at org.jboss.mq.server.JMSServerInterceptorSupport.acknowledge(JMSServerInterceptorSupport.java:197)

      at org.jboss.mq.server.TracingInterceptor.acknowledge(TracingInterceptor.java:405)

      at org.jboss.mq.server.JMSServerInvoker.acknowledge(JMSServerInvoker.java:199)

      at org.jboss.mq.il.uil2.ServerSocketManagerHandler.handleMsg(ServerSocketManagerHandler.java:81)

      at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleMsg(SocketManager.java:355)

      at org.jboss.mq.il.uil2.msgs.BaseMsg.run(BaseMsg.java:377)

      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:732)

      at java.lang.Thread.run(Thread.java:479)

      Java Stack for "Thread-184":

      ==========

      at org.jboss.mq.server.BasicQueue.hasUnackedMessages(BasicQueue.java:500)

      at org.jboss.mq.server.BasicQueue.removeSubscriber(BasicQueue.java:178)

      at org.jboss.mq.server.JMSQueue.removeSubscriber(JMSQueue.java:60)

      at org.jboss.mq.server.ClientConsumer.removeSubscription(ClientConsumer.java:253)

      at org.jboss.mq.server.JMSDestinationManager.unsubscribe(JMSDestinationManager.java:624)

      at org.jboss.mq.server.JMSServerInterceptorSupport.unsubscribe(JMSServerInterceptorSupport.java:249)

      at org.jboss.mq.server.TracingInterceptor.unsubscribe(TracingInterceptor.java:548)

      at org.jboss.mq.server.JMSServerInvoker.unsubscribe(JMSServerInvoker.java:251)

      at org.jboss.mq.il.uil2.ServerSocketManagerHandler.handleMsg(ServerSocketManagerHandler.java:171)

      at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleMsg(SocketManager.java:355)

      at org.jboss.mq.il.uil2.msgs.BaseMsg.run(BaseMsg.java:377)

      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:732)

      at java.lang.Thread.run(Thread.java:479)



      Found 1 deadlock.

        • 1. Re: Hey - Adrian bug [ 745527 ] race condition not fixed
          john Newbie

          Just an FYI - each synch causes four byte code instruction and you have synch code that has four syncs. this will generate 16 byte code instruction. Sychronizing on the method will reduce this by 75%

          • 2. Re: Hey - Adrian bug [ 745527 ] race condition not fixed
            Tameshwar Sahu Newbie

            Hi all,

            I am also getting the same error. We have JBoss running under HP-UX 11.11. When we try to send message in high speed to the queue and receive the message in same speed. After 1-2 hours all the receiver threads hangs and in jboss console , it displays the error:

            "CPMessageDistributor3" prio=10 tid=0x00575ae0 nid=1107 lwp_id=4869387 waiting for monitor entry [0x4d713000..0x4d713530]
            at org.jboss.mq.server.BasicQueue.receive(BasicQueue.java:238)
            - waiting to lock <0x591835f8> (a java.util.HashSet)
            at org.jboss.mq.server.JMSQueue.receive(JMSQueue.java:130)
            at org.jboss.mq.server.ClientConsumer.receive(ClientConsumer.java:225)
            at org.jboss.mq.server.JMSDestinationManager.receive(JMSDestinationManager.java:672)
            at org.jboss.mq.server.JMSServerInterceptorSupport.receive(JMSServerInterceptorSupport.java:225)
            at org.jboss.mq.security.ServerSecurityInterceptor.receive(ServerSecurityInterceptor.java:103)
            at org.jboss.mq.server.TracingInterceptor.receive(TracingInterceptor.java:478)
            at org.jboss.mq.server.JMSServerInvoker.receive(JMSServerInvoker.java:227)
            at org.jboss.mq.il.jvm.JVMServerIL.receive(JVMServerIL.java:245)
            at org.jboss.mq.Connection.receive(Connection.java:1166)
            at org.jboss.mq.SpyMessageConsumer.receive(SpyMessageConsumer.java:266)
            - locked <0x6001e0f0> (a java.util.LinkedList)
            at com.mobilgw.sam.executionengine.queue.CPMessageDistributor.run(CPMessageDistributor.java:153)
            at java.lang.Thread.run(Thread.java:536)

            Pleaae give your valuable suggestions.

            Thanks in advance.

            Monu

            • 3. Re: Hey - Adrian bug [ 745527 ] race condition not fixed
              Adrian Brock Master

              I simplified the synchronization for jboss-3.2.2RC1
              here is the diff
              http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/jboss/jbossmq/src/main/org/jboss/mq/server/BasicQueue.java.diff?r1=1.20.2.10&r2=1.20.2.11

              The comments about bytecode instructions are
              illconceived.

              Replacing a few bytecodes with a global synchronization
              will only slow things down as everything waits on a
              single point of contention.

              Regards,
              Adrian

              • 4. Re: Hey - Adrian bug [ 745527 ] race condition not fixed
                Tameshwar Sahu Newbie

                Adrian, Thank you very much.

                I am using Jboss3.2.1 and facing the reported problem. I think Jboss3.2.2.RCx is not ready for production. How can I use the updated BasicQueue class with JBoss3.2.1.
                Please give your valuable suggestions. I am facing very big problem because of deadlock issue.

                Thanks,
                Monu

                • 5. Re: Hey - Adrian bug [ 745527 ] race condition not fixed
                  Adrian Brock Master

                  You can get the 3.2.1 source and apply the diff
                  I pointed to.

                  Regards,
                  Adrian

                  • 6. Re: Hey - Adrian bug [ 745527 ] race condition not fixed
                    Tameshwar Sahu Newbie

                    Adrin, Thanx so much.
                    I tried to put chnages in BasicQueue.java of Jboss3.2.1 source but it seems there is a big difference in between jboss3.2.2 RC2 and jboss3.2.1 BasicQueue.java, that is why i clould not merge these properly. Could you please send me the BasicQueue.java in which you have fixed the deadlock bug.

                    Thanks,
                    Monu

                    • 7. Re: Hey - Adrian bug [ 745527 ] race condition not fixed
                      Adrian Brock Master

                      The version where I fixed is the link I provided.

                      There were other changes around the same time
                      for scheduled delivery that probably make it difficult
                      to apply.

                      Why don't you download jbossmq-3.2.2RC4
                      take jbossmq.jar and jbossmq-client.jar from it.
                      As far as I'm concerned that is stable (even if the
                      rest of jboss in the release candidate still has issues).

                      NOTE: If you use jbossall-client.jar you'll need
                      to update the org/jboss/mq classes.

                      Regards,
                      Adrian