7 Replies Latest reply on May 5, 2014 10:20 AM by Swapnil Bharate

    Can't reconnect to JMS server after cluster node goes down

    knaan ratosh Newbie

      Hi,
      I'm running jboss in cluster configuration - 2 nodes on the same machine with different ports.
      I have a queue to which messages are sent.
      When i stop the master node, i immediately see errors in the log. Here are the main log messages i see:
      1. connection to the JMS server could not be disabled
      2. Trying to reconnect to JMS provider --> Reconnected to JMS provider
      3. Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error and reconnect org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable:
      (java.io.IOException: Client is not connected)
      4. When trying to send messages to the queue - Cannot send a message to the JMS server; - nested throwable:
      (java.io.IOException: Client is not connected)

      Any idea how to solve this problem?
      Can it be only when the nodes are installed on the same machine?

      Here is the full log:
      2007-01-22 08:57:47,069 WARN [org.jboss.ejb.plugins.jms.JMSContainerInvoker] JMS provider

      failure detected:
      org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable: (java.io.EOFException)
      at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
      at org.jboss.mq.Connection.asynchFailure(Connection.java:437)
      at org.jboss.mq.il.uil2.UILClientILService.asynchFailure

      (UILClientILService.java:156)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:413)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:345)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.readByte

      (ObjectInputStream.java:2670)
      at java.io.ObjectInputStream.readByte(ObjectInputStream.java:864)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:290)
      ... 1 more
      2007-01-22 08:57:47,074 ERROR [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Could not

      stop JMS connection
      org.jboss.mq.SpyJMSException: Cannot disable the connection with the JMS server; - nested

      throwable: (java.io.IOException: Client is not connected)
      at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
      at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
      at org.jboss.mq.Connection.doStop(Connection.java:1235)
      at org.jboss.mq.Connection.stop(Connection.java:696)
      at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerStopDelivery

      (JMSContainerInvoker.java:987)
      at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl.run

      (JMSContainerInvoker.java:1447)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: java.io.IOException: Client is not connected
      at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
      at org.jboss.mq.il.uil2.SocketManager.sendMessage(SocketManager.java:206)
      at org.jboss.mq.il.uil2.UILServerIL.setEnabled(UILServerIL.java:197)
      at org.jboss.mq.Connection.doStop(Connection.java:1231)
      ... 4 more
      2007-01-22 08:57:47,076 WARN [org.jboss.mq.Connection] Connection failure, use

      javax.jms.Connection.setExceptionListener() to handle this error and reconnect
      org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable: (java.io.EOFException)
      at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
      at org.jboss.mq.Connection.asynchFailure(Connection.java:437)
      at org.jboss.mq.il.uil2.UILClientILService.asynchFailure

      (UILClientILService.java:156)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:413)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:345)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.readByte

      (ObjectInputStream.java:2670)
      at java.io.ObjectInputStream.readByte(ObjectInputStream.java:864)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:290)
      ... 1 more
      2007-01-22 08:57:47,079 WARN [org.jboss.mq.Connection] Connection failure, use

      javax.jms.Connection.setExceptionListener() to handle this error and reconnect
      org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable: (java.io.EOFException)
      at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
      at org.jboss.mq.Connection.asynchFailure(Connection.java:437)
      at org.jboss.mq.il.uil2.UILClientILService.asynchFailure

      (UILClientILService.java:156)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:413)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:345)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataI.mq.il.uil2.SocketManager$ReadTask.run

      (SocketManager.java:290)
      ... 1 more
      2007-01-22 08:57:57,100 INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to

      reconnect to JMS provider
      2007-01-22 08:57:57,345 INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected

      to JMS provider
      2007-01-22 08:57:58,540 WARN [org.jboss.mq.Connection] Connection failure, use

      javax.jms.Connection.setExceptionListener() to handle this error and reconnect
      org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable:

      (java.io.IOException: Client is not connected)
      at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
      at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
      at org.jboss.mq.Connection.pingServer(Connection.java:887)
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1327)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: java.io.IOException: Client is not connected
      at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
      at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:224)
      at org.jboss.mq.il.uil2.UILServerIL.ping(UILServerIL.java:470)
      at org.jboss.mq.Connection.pingServer(Connection.java:883)
      ... 3 more
      2007-01-22 08:58:02,800 WARN [org.jboss.mq.Connection] Connection failure, use

      javax.jms.Connection.setExceptionListener() to handle this error and reconnect
      org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable:

      (java.io.IOException: Client is not connected)
      at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
      at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
      at org.jboss.mq.Connection.pingServer(Connection.java:887)
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1327)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: java.io.IOException: Client is not connected
      at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
      at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:224)
      at org.jboss.mq.il.uil2.UILServerIL.ping(UILServerIL.java:470)
      at org.jboss.mq.Connection.pingServer(Connection.java:883)
      ... 3 more
      2007-01-22 08:58:49,210 INFO

      [messaging_as.module.mail.conf.MASMailConnectionCleaningScheduler] --- Starting to clean

      idle connections ---
      2007-01-22 08:58:49,211 INFO

      [messaging_as.module.mail.conf.MASMailConnectionCleaningScheduler] Connection cache has [0]

      items, [0] of them are idle.
      2007-01-22 08:58:49,211 INFO

      [messaging_as.module.mail.conf.MASMailConnectionCleaningScheduler] --- Idle connections

      clean completed ---
      2007-01-22 08:58:58,550 WARN [org.jboss.mq.Connection] Connection failure, use

      javax.jms.Connection.setExceptionListener() to handle this error and reconnect
      org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException:

      ping timeout.)
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1323)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: java.io.IOException: ping timeout.
      ... 3 more

      AND THEN MORE:
      53 2007-01-22 09:06:59,087 DEBUG [messaging_as.module.poller.service.DispatcherBean]

      Account info as string: 753;8455400000;OPTIMIZER;null;null;null;Israel Standard

      Time;403;8455400000;1234;IMAP;143;10.14.1.40;IMAP_IDLE;510;NoFilter;iso-8859-1
      753 2007-01-22 09:06:59,087 ERROR [messaging_as.module.poller.service.DispatcherBean] Cannot

      send a message to the JMS server; - nested throwable: (java.io.IOException: Client is not

      connected)
      org.jboss.mq.SpyJMSException: Cannot send a message to the JMS server; - nested throwable:

      (java.io.IOException: Client is not connected)
      at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
      at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
      at org.jboss.mq.Connection.sendToServer(Connection.java:969)
      at org.jboss.mq.SpySession.sendMessage(SpySession.java:975)
      at org.jboss.mq.SpyMessageProducer.send(SpyMessageProducer.java:265)
      at org.jboss.mq.SpyMessageProducer.send(SpyMessageProducer.java:199)
      at

      messaging_as.module.poller.service.DispatcherBean.putMessageWithPollAccountInfoInQueue

      (DispatcherBean.java:262)