7 Replies Latest reply on May 5, 2014 10:20 AM by sbharate

Can't reconnect to JMS server after cluster node goes down

kratosh Jan 23, 2007 7:54 AM

Hi,
I'm running jboss in cluster configuration - 2 nodes on the same machine with different ports.
I have a queue to which messages are sent.
When i stop the master node, i immediately see errors in the log. Here are the main log messages i see:
1. connection to the JMS server could not be disabled
2. Trying to reconnect to JMS provider --> Reconnected to JMS provider
3. Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error and reconnect org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable:
(java.io.IOException: Client is not connected)
4. When trying to send messages to the queue - Cannot send a message to the JMS server; - nested throwable:
(java.io.IOException: Client is not connected)

Any idea how to solve this problem?
Can it be only when the nodes are installed on the same machine?

Here is the full log:
2007-01-22 08:57:47,069 WARN [org.jboss.ejb.plugins.jms.JMSContainerInvoker] JMS provider

failure detected:
org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable: (java.io.EOFException)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.Connection.asynchFailure(Connection.java:437)
at org.jboss.mq.il.uil2.UILClientILService.asynchFailure

(UILClientILService.java:156)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:413)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:345)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.readByte

(ObjectInputStream.java:2670)
at java.io.ObjectInputStream.readByte(ObjectInputStream.java:864)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:290)
... 1 more
2007-01-22 08:57:47,074 ERROR [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Could not

stop JMS connection
org.jboss.mq.SpyJMSException: Cannot disable the connection with the JMS server; - nested

throwable: (java.io.IOException: Client is not connected)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
at org.jboss.mq.Connection.doStop(Connection.java:1235)
at org.jboss.mq.Connection.stop(Connection.java:696)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerStopDelivery

(JMSContainerInvoker.java:987)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl.run

(JMSContainerInvoker.java:1447)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
at org.jboss.mq.il.uil2.SocketManager.sendMessage(SocketManager.java:206)
at org.jboss.mq.il.uil2.UILServerIL.setEnabled(UILServerIL.java:197)
at org.jboss.mq.Connection.doStop(Connection.java:1231)
... 4 more
2007-01-22 08:57:47,076 WARN [org.jboss.mq.Connection] Connection failure, use

javax.jms.Connection.setExceptionListener() to handle this error and reconnect
org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable: (java.io.EOFException)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.Connection.asynchFailure(Connection.java:437)
at org.jboss.mq.il.uil2.UILClientILService.asynchFailure

(UILClientILService.java:156)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:413)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:345)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.readByte

(ObjectInputStream.java:2670)
at java.io.ObjectInputStream.readByte(ObjectInputStream.java:864)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:290)
... 1 more
2007-01-22 08:57:47,079 WARN [org.jboss.mq.Connection] Connection failure, use

javax.jms.Connection.setExceptionListener() to handle this error and reconnect
org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable: (java.io.EOFException)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.Connection.asynchFailure(Connection.java:437)
at org.jboss.mq.il.uil2.UILClientILService.asynchFailure

(UILClientILService.java:156)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:413)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:345)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataI.mq.il.uil2.SocketManager$ReadTask.run

(SocketManager.java:290)
... 1 more
2007-01-22 08:57:57,100 INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to

reconnect to JMS provider
2007-01-22 08:57:57,345 INFO [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected

to JMS provider
2007-01-22 08:57:58,540 WARN [org.jboss.mq.Connection] Connection failure, use

javax.jms.Connection.setExceptionListener() to handle this error and reconnect
org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable:

(java.io.IOException: Client is not connected)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
at org.jboss.mq.Connection.pingServer(Connection.java:887)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1327)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:224)
at org.jboss.mq.il.uil2.UILServerIL.ping(UILServerIL.java:470)
at org.jboss.mq.Connection.pingServer(Connection.java:883)
... 3 more
2007-01-22 08:58:02,800 WARN [org.jboss.mq.Connection] Connection failure, use

javax.jms.Connection.setExceptionListener() to handle this error and reconnect
org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable:

(java.io.IOException: Client is not connected)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
at org.jboss.mq.Connection.pingServer(Connection.java:887)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1327)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:224)
at org.jboss.mq.il.uil2.UILServerIL.ping(UILServerIL.java:470)
at org.jboss.mq.Connection.pingServer(Connection.java:883)
... 3 more
2007-01-22 08:58:49,210 INFO

[messaging_as.module.mail.conf.MASMailConnectionCleaningScheduler] --- Starting to clean

idle connections ---
2007-01-22 08:58:49,211 INFO

[messaging_as.module.mail.conf.MASMailConnectionCleaningScheduler] Connection cache has [0]

items, [0] of them are idle.
2007-01-22 08:58:49,211 INFO

[messaging_as.module.mail.conf.MASMailConnectionCleaningScheduler] --- Idle connections

clean completed ---
2007-01-22 08:58:58,550 WARN [org.jboss.mq.Connection] Connection failure, use

javax.jms.Connection.setExceptionListener() to handle this error and reconnect
org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException:

ping timeout.)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1323)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: ping timeout.
... 3 more

AND THEN MORE:
53 2007-01-22 09:06:59,087 DEBUG [messaging_as.module.poller.service.DispatcherBean]

Account info as string: 753;8455400000;OPTIMIZER;null;null;null;Israel Standard

Time;403;8455400000;1234;IMAP;143;10.14.1.40;IMAP_IDLE;510;NoFilter;iso-8859-1
753 2007-01-22 09:06:59,087 ERROR [messaging_as.module.poller.service.DispatcherBean] Cannot

send a message to the JMS server; - nested throwable: (java.io.IOException: Client is not

connected)
org.jboss.mq.SpyJMSException: Cannot send a message to the JMS server; - nested throwable:

(java.io.IOException: Client is not connected)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
at org.jboss.mq.Connection.sendToServer(Connection.java:969)
at org.jboss.mq.SpySession.sendMessage(SpySession.java:975)
at org.jboss.mq.SpyMessageProducer.send(SpyMessageProducer.java:265)
at org.jboss.mq.SpyMessageProducer.send(SpyMessageProducer.java:199)
at

messaging_as.module.poller.service.DispatcherBean.putMessageWithPollAccountInfoInQueue

(DispatcherBean.java:262)

1. Re: Can't reconnect to JMS server after cluster node goes do

brian.stansberry Jan 23, 2007 12:16 PM (in response to kratosh)

Is this from an MDB or your own JMS client? If your own, please have a look at http://wiki.jboss.org/wiki/Wiki.jsp?page=IGetIOExceptionPingTimeoutHowDoIFixIt.

If from an MDB, what AS version? I know in some older versions some of the messages that were logged during HA-JMS failover while the MDB reconnected were logged at too high a level. If the MDB eventually reconnects, this may be the issue.
Actions
2. Re: Can't reconnect to JMS server after cluster node goes do

kratosh Jan 23, 2007 5:01 PM (in response to kratosh)

I'm using a QueueSender in some class of mine in order to send messages to the queue. And i use a QueueConnection that is created by QueueConnectionFactory.
I use MDB for getting messages from the queue, not sending to the queue. And the problem is with sending the messages.
I looked at the wiki page you gave me, but i think they only explain how to get rid of the errors in the log. while i see in the log that the JMS server reconnected, but then when i try to send a message to it it fails - that means, i think, that the queue is not connected.
What else can i do?

I use jboss4.0.3.

Another thing i noticed is that when the 2 nodes are on different machines, the failover is done correctly, and after the reconnection the message sending succeeds. I see the problem only when the 2 nodes are on the same machine... strange... any idea how it can be related to that?

Also, i tried to stop and start my queue through JMX-console and it didn't help.
Actions
3. Re: Can't reconnect to JMS server after cluster node goes do

brian.stansberry Jan 24, 2007 1:08 PM (in response to kratosh)

Are you re-looking up the queue and recreating the Session and MessageProducer?
Actions
4. Re: Can't reconnect to JMS server after cluster node goes do

kratosh Jan 28, 2007 3:41 AM (in response to kratosh)

In the ejbRemove() method i do _queueConnection.close();
In the ejbCreate() method i lookup the quque and create the session.
Here is the relevant code from ejbCreate():

_queueConnectionFactory = (QueueConnectionFactory) _jndiContext.lookup("ConnectionFactory");
_queue = (Queue) _jndiContext.lookup(_queueName);
_queueConnection =_queueConnectionFactory.createQueueConnection();
_queueSession =
_queueConnection.createQueueSession(false,
Session.AUTO_ACKNOWLEDGE);
_queueSender = _queueSession.createSender(_queue);
Actions
5. Re: Can't reconnect to JMS server after cluster node goes do

brian.stansberry Jan 28, 2007 12:38 PM (in response to kratosh)
Does using

_jndiContext.lookup("java:/JmsXA");

resolve your issue?

Otherwise, you need to register an exception listener with the connection, use it to get a callback when an exception occurs on the connection, and in the callback re-lookup the connection factory and queue and reestablish the connection, session and sender.
Actions
6. Re: Can't reconnect to JMS server after cluster node goes do

brian.stansberry Jan 28, 2007 2:14 PM (in response to kratosh)

http://wiki.jboss.org/wiki/Wiki.jsp?page=JMSConnectionExceptionListener provides more details.
Actions
7. Re: Can't reconnect to JMS server after cluster node goes down

sbharate May 5, 2014 10:20 AM (in response to brian.stansberry)

Hey,

Yes I got also the same issue as :

WARN [org.jboss.mq.Connection] Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error and reconnect
org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)

Caused by: java.io.IOException: ping timeout.

Will you please help me out of this ??

Many many thanks to you.
Actions

Go to original post