7 Replies Latest reply on Feb 20, 2004 8:42 PM by Priyapravas Avasthi

    JMS hangs after IOException in the server

    Richard Bielak Newbie

      We're running Jboss 3.2.3 on Linux (JDK 1.4.1). Our application uses publishes messages on several topics (non-persistent messages). The load is pretty heavy - we have about 40 clients publishing and receiving messages every 10 seconds.

      Few times the whole JMS just hanged, after an exception in server. The IO exception was from "internalSend" in org.jboss.mq.il.uil2.SocketManager and it said:

      "Client not connected"

      After the initial exception eveything just stops. Except the same exception keeps being thrown in the server log. The clients don't seem to get any exceptions at all.

      Restarting the clients has no effect and only restarting JBoss fixes the problem.

      The JMS is configured to use file based persistance manager, as we normally do not expect message to hang around.

      I'm unable to post extract from the log at this point, because these are production machines and getting logs copied over gets difficult.

      Does this problem sound familiar to anyone? Any suggestions where to look will be greatly appreciated.

      What other information would help in finding what the problem is?

      TIA

      ...richie

        • 1. Re: JMS/MDB JBoss 4.0.0 on mySQL
          Priyapravas Avasthi Newbie

          It is probably easier and better to look for a suitable database file in the doc directory of the jboss install.

          • 2. Re: JMS hangs after IOException in the server
            Adrian Brock Master

            The server should disconnect the client when it cannot send a message
            to the client. I would need to see the logs to find out what is going on.

            Regards,
            Adrian

            • 3. Re: JMS hangs after IOException in the server
              Richard Bielak Newbie

              Here is a small extract from the server log:

              2004-02-20 18:26:10,946 ERROR [org.jboss.mq.server.JMSDestinationManager] The connection to client ID
              :7 failed.
              2004-02-20 18:26:10,948 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to handle: org.jboss.mq.il.
              uil2.msgs.AcknowledgementRequestMsg1649195744[msgType: m_acknowledge, msgID: 171372, error: null]
              java.io.IOException: Client is not connected
              at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:225)
              at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:208)
              at org.jboss.mq.il.uil2.ServerSocketManagerHandler.handleMsg(ServerSocketManagerHandler.java:
              82)
              at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleMsg(SocketManager.java:355)
              at org.jboss.mq.il.uil2.msgs.BaseMsg.run(BaseMsg.java:377)
              at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:732)
              at java.lang.Thread.run(Thread.java:568)


              We see no exceptions on the client side.

              The client seems to be able to publish without errors, but does not see any new messages.

              I haven't been able to reproduce this problem with a small test case.

              ...richie

              • 4. Re: JMS hangs after IOException in the server
                Adrian Brock Master

                The first message says the server is going to close the connection.

                The second message says the client was still trying to acknowledge
                receipt of a message.

                Have you set an ExceptionListener on the connection so you can monitor for
                errors in the transport?

                Regards,
                Adrian

                • 5. Re: JMS hangs after IOException in the server
                  Richard Bielak Newbie

                  Yes we have. Here are snippets of our code:


                  tcf_ = (TopicConnectionFactory) context_.lookup(connectionFactory_);
                  conn_ = tcf_.createTopicConnection ();
                  conn_.setExceptionListener(new ExceptionListener() {
                  public void onException(JMSException e)
                  {
                  if (!restarting_)
                  JMSInterface.shutdownAndRestart(e);
                  }
                  });


                  In "shutdownAndRestart" we attempt to disconnect and reconnect to JMS. We log several messages - which I do not see in this case. Here are more snipets of the code:


                  private static void shutdownAndRestart(Throwable t) {

                  TSLogger.getMainLogger().debug("JMS Service Failure", t);
                  restarted_ = false;
                  synchronized (JMSInterface.class) {
                  if (restarted_)
                  return;
                  restarting_ = true;
                  for (;;) {
                  TSLogger.getMainLogger().warn("JMS service not reachable ... will try again in a few seconds");
                  try {
                  Thread.sleep (reconnectTimeout_);
                  }
                  catch (InterruptedException ie) {}

                  ... etc


                  We don not see the warning message. In a more typical operation, when we shut down JBoss we see the warnings logged by the clients. When JBoss is restarted the clients reconnect and continue.

                  ...richie


                  • 6. Re: JMS hangs after IOException in the server
                    Adrian Brock Master

                    Since you didn't get a disconnect message on the client it sounds
                    like the server failed to close the connection.

                    Unfortunatly, the logging of an error from this operation has been disabled
                    // Close the clientIL
                    try
                    {
                    if (dc.clientIL != null)
                    dc.clientIL.close();
                    }
                    catch (Exception ex)
                    {
                    // We skipp warning, to often the client will allways
                    // have gone when we get here
                    //log.warn("Could not close clientIL: " +ex,ex);
                    }

                    Another option is that it got stuck in a deadlock before hitting this code.
                    This would be consistent with you saying "JMS hangs".
                    I don't suppose you took a thread dump while it hung?

                    Regards,
                    Adrian

                    • 7. Re: JMS hangs after IOException in the server
                      Richard Bielak Newbie

                      I'm afraid I don't have a thread dump. This is a production system, so I only find out after a JBoss restart.


                      ...richie