Slight problem still regarding the fix for bug JBAS-1489

[jmihalich]

Hi,

We were experiencing the issue described in bug 1489. We successfully patched our system with the files that were fixed for this bug, patching just jbossmq.jar.

In our case, the thing that caused the JMS server's socket to die was someone pulling the network cable from the network card on our box. Now, when the cable gets plugged back in, everything reconnects as designed with the fix.

Howevere, there remains one issue. Even though the connections are re-established, we still get this exception repeatitively:

2005-08-05 15:01:30,559 [hread] WARN org.jboss.mq.Connection : Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error and reconnect
org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1307)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.io.IOException: ping timeout.
... 3 more
2005-08-05 15:01:30,700 [hread] WARN org.jboss.mq.Connection : Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error and reconnect
org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1307)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.io.IOException: ping timeout.
... 3 more


Prior to those messages continuing, I see this message for all topics that are running on our server:

2005-08-05 14:48:56,655 [hread] INFO org.jboss.ejb.plugins.jms.JMSContainerInvoker : Reconnected to JMS provider


Is this a bug that is still lingering? It's not harming anything that I can notice, but this should go away once all connections are re-established right?


Thanks,
Joe

[adrian.brock]

"jmihalich" wrote:

2005-08-05 14:48:56,655 [hread] INFO org.jboss.ejb.plugins.jms.JMSContainerInvoker : Reconnected to JMS provider

JBAS-1489 is completely unrelated to MDBs. It is for user created MessageListeners.

Useful links:
http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossHelp
and
READ THIS FIRST

2005-08-05 15:01:30,559 [hread] WARN org.jboss.mq.Connection : Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error and reconnect

It is almost worth adding a Stacktrace of where the connection was allocated
to this error message, then I wouldn't have to ask for the TRACE logging all the time
(not that I do anymore, I just ignore people that cannot READ THIS FIRST).

P.S. I'm only interested in this problem if you can show the connection that is still
trying to ping is from the MDB or java:/JmsXA
AND it is still a problem in the latest release.

[shadkam]

Hi,

Even we are facing the same problem mentioned by Joe. But for us, the same code works fine on BSD and Linux, but does not work on windows XP.
Following is the scenario:
o Server is up, happily posting msgs to the clients, everything is fine.
o Somebody pulls out the network cable of the server machine
o Understandably, it is no longer able to send any msgs, and it shows Exceptions pasted below.
o Now, even when the network cable is connected back, other things (like request response / session EJB calls) work fine, but the JMS never seems to recover. It keeps on printing the same Exceptions for every call to publish method.

19:50:00,682 WARN [Connection] Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error
and reconnect
org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable: (java.io.IOException: Client is not connected)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
at org.jboss.mq.Connection.pingServer(Connection.java:887)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1327)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:224)
at org.jboss.mq.il.uil2.UILServerIL.ping(UILServerIL.java:462)
at org.jboss.mq.Connection.pingServer(Connection.java:883)
... 3 more
19:50:00,698 WARN [Connection] Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error
and reconnect
org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable: (java.io.IOException: Client is not connec
ted)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
at org.jboss.mq.Connection.pingServer(Connection.java:887)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1327)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:224)
at org.jboss.mq.il.uil2.UILServerIL.ping(UILServerIL.java:462)
at org.jboss.mq.Connection.pingServer(Connection.java:883)
... 3 more
19:50:00,745 WARN [Connection] Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error
and reconnect
org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable: (java.io.IOException: Client is not connected)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
at org.jboss.mq.Connection.pingServer(Connection.java:887)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1327)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:224)
at org.jboss.mq.il.uil2.UILServerIL.ping(UILServerIL.java:462)
at org.jboss.mq.Connection.pingServer(Connection.java:883)
... 3 more
19:50:00,761 WARN [Connection] Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error
and reconnect
org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable: (java.io.IOException: Client is not connected)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
at org.jboss.mq.Connection.pingServer(Connection.java:887)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1327)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:224)
at org.jboss.mq.il.uil2.UILServerIL.ping(UILServerIL.java:462)
at org.jboss.mq.Connection.pingServer(Connection.java:883)
... 3 more

[shadkam]

Also, I forgot to mention one thing ... we did try setting exceptionListener and then handle the exception, we re-got the ConnectionFactory from JNDI and created the sessions etc afresh ... but that did not help, perhaps because the ConnectionFactory itself had become corrupted (its internal sockets) ... apart from redeployment, is there a way to correct those ...
thanks
shadkam

[shadkam]

Also, I forgot to mention one thing ... we did try setting exceptionListener and then handle the exception, we re-got the ConnectionFactory from JNDI and created the sessions etc afresh ... but that did not help, perhaps because the ConnectionFactory itself had become corrupted (its internal sockets) ... apart from redeployment, is there a way to correct those ...
thanks
shadkam

[jokum]

Any thought on this? I'm currently facing the exact same problem. (Windows XP)

[talktoudaykumar]

I'm facing the same problem. I don't know what went wrong in that. Please help me in this context. It's Urjent Please......Please....Please....

[adrian.brock]

Which part of

use javax.jms.Connection.setExceptionListener() to handle this error
and reconnect

don't you guys understaand?

[beefcaaaake]

Well Adrian, that is exactly what I did. When I initialize queue, I register for exception callbacks...

conn = qcf.createQueueConnection();
// Register for exception callbacks
conn.setExceptionListener(new ExceptionListenerImpl());
session = conn.createQueueSession(false, QueueSession.AUTO_ACKNOWLEDGE);

I got the idea from here...

http://wiki.jboss.org/wiki/JMSConnectionExceptionListener

Looks pretty straight forward. So after implementing the exception listener and redeploying, I still get the failure below.

10:42:08,781 WARN [Connection] Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error
 and reconnect
org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
 at org.jboss.mq.Connection$PingTask.run(Connection.java:1313)
 at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
 at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: ping timeout.
 ... 3 more

Interesting this is that I have added logging to the callback implementation and none of it shows up on the console. This tells me that the call back isn't being called.

Am I doing something wrong? It all started when I lost network connectivity for a few seconds. Also, my jboss instance is bound to the ip address of the machine it is running on, instead of the localhost default.

[adrian.brock]

"beefcaaaake" wrote:
Well Adrian, that is exactly what I did.

No you didn't, otherwise you wouldn't get that error message.

Try

conn = qcf.createQueueConnection();
// Register for exception callbacks
conn.setExceptionListener(new ExceptionListenerImpl());
System.out.println("=========> Now running the new code");
session = conn.createQueueSession(false, QueueSession.AUTO_ACKNOWLEDGE);

There's also an FAQ on how to track down where you forgot to do it.

Usually when this message comes up, people swear blind they've done
it correctly and then find they're creating a connection somewhere else
which doesn't have the exception listener. :-)

[kevinjm]

We are experiencing a very similar problem, and I was hoping on clarification on whether or not there is a bug here, and if it has been fixed. Or if there is a way to debug this issue.

The server is 4.0.3SP1 jboss server on a linux box. The client is 4.0.2.

Our JMS client has a topic connection that has a listener. The listener sets a flag, and another thread will [very quickly] clean up and then reconnect when it sees the flag set.

The problems began when there was a network outage.

Since then, we've been in a cycling loop of reconnects. The client gets a "no pong received". The onException is invoked. The thread cleans up and reconnects. Messages are transferred from the client to the server, and from the server to the client. A minute later, we get a "no pong received" exception - repeat.

2008-08-06 09:46:01,416 WARN [TopicConnectionManager] the connection to the management server was broken: org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
2008-08-06 09:46:02,112 WARN [TopicConnectionManager] successfully established a connection to the management server
2008-08-06 09:47:01,416 WARN [TopicConnectionManager] the connection to the management server was broken: org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
2008-08-06 09:47:02,459 WARN [TopicConnectionManager] successfully established a connection to the management server
2008-08-06 09:48:01,420 WARN [TopicConnectionManager] the connection to the management server was broken: org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
2008-08-06 09:48:01,811 WARN [TopicConnectionManager] successfully established a connection to the management server
2008-08-06 09:49:01,424 WARN [TopicConnectionManager] the connection to the management server was broken: org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
2008-08-06 09:49:02,152 WARN [TopicConnectionManager] successfully established a connection to the management server
2008-08-06 09:50:01,424 WARN [TopicConnectionManager] the connection to the management server was broken: org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
2008-08-06 09:50:01,500 WARN [TopicConnectionManager] successfully established a connection to the management server
2008-08-06 09:51:01,428 WARN [TopicConnectionManager] the connection to the management server was broken: org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)

Here's what the debug log shows.

2008-08-06 10:20:01,476 DEBUG [TopicConnectionManager] exception reported on topic connection
org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
 at org.jboss.mq.Connection$PingTask.run(Connection.java:1323)
 at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
 at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: ping timeout.
 ... 3 more

We have seen the same exact problem on many clients on different machines, connected to the same server. The log file shows a "no pong received" exception delivered every 60 seconds, apparently forever.

There are no huge messages, or large numbers of exceptions. This is verified with tcpdump.

Since JMS seems like its otherwise working - server sees what the client published, client sees what the server published - this looks like some sort of JBoss issue.

Can anyone help? I'm not sure what else it would be helpful to know.

[javacory]

We are experiencing this problem as well. The result of this issue is we eventually receive a "too many open files" error. I assume this is the result of the "no pong received" which prompts the server to create a new connection. We can see the established sockets ( 100s of them ) before this error occurs using "netstat -an".

Our environment is JBoss 4.2.0GA. We have a JBoss server running which contains our JMS queues and one that connects remotely to those queues. If the server which contains the queues is restarted, we randomly get the ping error. It doesn't happen every time. I assume there is a timing issue here as to what state the server is in that has the queues and when the remote server tries to ping it.

In looking at the org.jboss.mq.Connection code, when this situation occurs, the "PingTask" runnable never gets to the "pingServer()" call because the "ponged" variable is always false. If the ponged variable is false, an exception is thrown causing it to never get to the "pingServer()".

Has this been fixed in a latter release of JBoss?

Thanks,

Cory.

[jmihalich]

I gave up trying to fix this problem. It didn't seem to be affecting anything, as you said. It just appears there's a thread out there that is still running that should have exited.

We switched to jboss messaging about 4 months ago, on jboss 4.2.2. All the problems we had with jboss mq with jboss mq deadlocking on heavy load, and this pong stuff went away. The new db schema and code they're using is so much better and more stable.

Joe