6 Replies Latest reply on Dec 3, 2003 11:05 AM by gmccreath

    JBossMQ Hanging

    ioparra

      We were recently testing how solid or stand alone product communicates with JMS. We did this by turning on/off our product both normally and brutally(ie. don't allow to clean up, just STOP NOW!).

      What we found was interesting. Everytime we killed a JMSClient, we would recieve a stack trace like this:

      16:41:55,674 WARN [OILServerILService] Connection failure (1).
      java.net.SocketException: Connection reset by peer: JVM_recv in socket input stream read
      at java.net.SocketInputStream.socketRead(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:85)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:181)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:199)
      at java.io.ObjectInputStream.peekCode(ObjectInputStream.java:1545)
      at java.io.ObjectInputStream.refill(ObjectInputStream.java:1679)
      at java.io.ObjectInputStream.read(ObjectInputStream.java:1655)
      at java.io.ObjectInputStream.readByte(ObjectInputStream.java:1901)
      at org.jboss.mq.il.oil.OILServerILService$Client.run(OILServerILService.java:205)
      at java.lang.Thread.run(Thread.java:479)
      16:41:55,674 WARN [OILServerILService] Connection failure (1).

      -----
      After doing this similar process a few times, the JMS Server acquires these characteristic:

      1) It is able to accept new JMS messages
      2) It does NOT process those messages to their MDB.

      Needless to say, the Queues start to grow.

      Attached are thread traces that show a dead lock scenario. It appears that a Message Pusher is waiting to read from a socket. At the same time, another thread (that which has a lock on the DestinationManager) is trying to close the socket.

      All other Workers who need access to the DestinationManager then have to wait for these threads to finish.(Which never happens).

      Attachment has a better explanation.

      Bug?

      -Ivan

      If you close your JMS Connection correctly, this case apears to be avoided. BUT any of your JMS clients fails to close correctly, this process may hang JMS.

        • 1. Re: JBossMQ Hanging
          ioparra

          i can't seem to attach files... URGH

          • 2. Re: JBossMQ Hanging
            ioparra

            Should of mentioned:

            JB3.2.1
            Win2KServer
            JDK1.3.1_06

            tested on JB3.2.2, same thing.

            • 3. Re: JBossMQ Hanging

              What you are seeing is the reason why all invocation layers
              except UIL2 are deprecated.

              The people that wrote the other invocation layers decided that
              putting synchronized on every method would solve all
              problems.

              The simple fix to this problem is to include a read timeout
              on the socket created in OILClientIL.

              Report or a bug at www.sourceforge.net or better still
              post a patch that fixes the problem.

              In the meantime, use UIL2ConnectionFactory.

              Regards,
              Adrian

              • 4. Re: JBossMQ Hanging
                gmccreath

                Hi Adrian,

                Does this mean the in-jvm il should also not be used?

                If I were using jms services inside a single jvm, that is piblishers and subscribers are all in-vm, should I change (for stability's sake) to UIL2 ?

                Greg.

                • 5. Re: JBossMQ Hanging

                  No the jvm-il is ok as is UIL2.
                  We will probably rewrite OIL to be more like UIL2 (i.e. no
                  stupid synchronizations)

                  Regards,
                  Adrian

                  • 6. Re: JBossMQ Hanging
                    gmccreath

                    Thanks Adrian, that is good news.