JMS Ping Timeout

[lucasfowler]

We are getting the following connection failed message whilst listening
on a topic.
The topic itself is ok, because if we connect a new client then it can read without
problems.

[2004-03-12 12:03:36,273] [WARN ] [org.jboss.mq.Connection] Connection failure:
org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Unknown Source)
at java.lang.Thread.run(Thread.java:536)
Caused by: java.io.IOException: ping timeout.
at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
... 2 more

I can provide a stack trace if required, but it is too big to post here

Here is the stdout.log:
================================================================================

JBoss Bootstrap Environment

JBOSS_HOME: /home/trdmdmo/AppServer/trdmdmo/JBossTomcat

JAVA: /opt2/JavaCentral/linux/j2sdk1.4.1_01/bin/java

JAVA_OPTS: -DJBOSS=602 -server -server -Xms4m -Xmx128m -Dprogram.name=run.sh

CLASSPATH: /home/trdmdmo/AppServer/trdmdmo/JBossTomcat/bin/run.jar:/opt2/JavaCentral/linux/j2sdk1.4.1_01/lib/tools.jar

================================================================================

06:08:33,140 INFO [Server] Starting JBoss (MX MicroKernel)...
06:08:33,144 INFO [Server] Release ID: JBoss [WonderLand] 3.2.3 (build: CVSTag=JBoss_3_2_3 date=200311301445)
06:08:33,154 INFO [Server] Home Dir: /home/trdmdmo/AppServer/trdmdmo/JBossTomcat
06:08:33,154 INFO [Server] Home URL: file:/home/trdmdmo/AppServer/trdmdmo/JBossTomcat/
06:08:33,155 INFO [Server] Library URL: file:/home/trdmdmo/AppServer/trdmdmo/JBossTomcat/lib/
06:08:33,158 INFO [Server] Patch URL: null
06:08:33,158 INFO [Server] Server Name: default
06:08:33,158 INFO [Server] Server Home Dir: /home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default
06:08:33,159 INFO [Server] Server Home URL: file:/home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default/
06:08:33,159 INFO [Server] Server Data Dir: /home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default/data
06:08:33,160 INFO [Server] Server Temp Dir: /home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default/tmp
06:08:33,160 INFO [Server] Server Config URL: file:/home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default/conf/
06:08:33,168 INFO [Server] Server Library URL: file:/home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default/lib/
06:08:33,168 INFO [Server] Root Deployment Filename: jboss-service.xml
06:08:33,193 INFO [Server] Starting General Purpose Architecture (GPA)...
06:08:33,899 INFO [ServerInfo] Java version: 1.4.1_01,Sun Microsystems Inc.
06:08:33,899 INFO [ServerInfo] Java VM: Java HotSpot(TM) Server VM 1.4.1_01-b01,Sun Microsystems Inc.
06:08:33,899 INFO [ServerInfo] OS-System: Linux 2.4.18-686-smp,i386
06:08:33,991 INFO [ServiceController] Controller MBean online
06:08:34,180 INFO [MainDeployer] Started jboss.system:service=MainDeployer
06:08:34,367 INFO [MainDeployer] Adding deployer: org.jboss.deployment.JARDeployer@1b09468
06:08:34,368 INFO [JARDeployer] Started jboss.system:service=JARDeployer
06:08:34,427 INFO [MainDeployer] Adding deployer: org.jboss.deployment.SARDeployer@861f24
06:08:34,453 INFO [SARDeployer] Started jboss.system:service=ServiceDeployer
06:08:34,454 INFO [Server] Core system initialized
06:08:34,508 INFO [MainDeployer] Starting deployment of package: file:/home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default/conf/jboss-service.xml

Can anyone give us some pointers?

Thanks

[adrian.brock]

javax.jms.Connection.setExceptionListener()

Regards,
Adrian

[mcofran]

I am currently experiencing the same issue that lucas has, but I don't understand what you're saying should be done. Are you suggesting we create an exception listener? What would it listen for and what would it do when it gets a message?

[chrisxsb]

I'm running into the same issue.
In other threads, it looks like the suggestion was to re-initialize the connection when you receive an exception to try to recover.

In my case, I have I have a temporary queue set up to receive a response from some message (that is, the reply-to field in some message I send is set to be my temporary queue). My understanding is that once a connection is closed, all temporary queues are destroyed -- how can I reinitialize the connection and have it receive messages on the temporary queue?

The only solution I can come up with is to not use temporary queues, but I'm hoping to avoid this, if possible.

Adrian? Anyone?

Thanks in advance.

[lucasfowler]

Thanks for the infos here.

We have added an exceptionListener and that seems to have done the trick.

[alanc_yang]

I encountered the same problem, but wasn't able to see the ExceptionListener's onException gets call when it happened. This was what I did:

1) in my MDBean to implement ExceptionListener
2) provide onException method
3) in the connection to setExceptionListener(this);

I must have missed something. Please help!
Thanks in advance.

Alan

[richieb]

"lucasfowler" wrote:
Thanks for the infos here.

We have added an exceptionListener and that seems to have done the trick.

And what does your listener do?

...richie

[tysmeister]

Hi Guys,

I am also experiencing this 'problem' with 3.2.3 and a Queue, however the bug appears to be harmless. Can the ExceptionHandler just ignore the Exception, or should it perform some cleanup ?

Thanks and regards,
Andrew

[alanc_yang]

I have two subscribers, one MBean and one MDBean that in 1 min apart, the exceptoin was thrown as in the following:

07:57:05,473 WARN [Connection] Connection failure:
org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException:
ping timeout.)
at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.io.IOException: ping timeout.
at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
... 2 more

Seems like the MDBean has no impact that it would continue receving the message. But the MBean failed. My MBean code was like this:

1. public class myMBean extends ServiceMBeanSupport ...
2. within startService(), it lanuches another thread
3. the other thread performs a while loop:
while (true) {
setupConnection();

msg = sub.receive();
processMsg(msg);

if(msg == null) System.out.println(" receiving alarm failed");
stopConnection();
}

The MBean would fail after the above exception. Would like to know is this the right way to code the message receiving loop, or how to work around the problem.

Thanks a lot.
Alan

[lucasfowler]

"richieb" wrote:

"lucasfowler" wrote:
Thanks for the infos here.

We have added an exceptionListener and that seems to have done the trick.

And what does your listener do?

...richie

OK, we are running in a servlet so we can't start another thread. So we
set up the listener to set a flag and changed the read method to timeout
every so often. That way, if there is a connection failure we can close
properly in our finally clause and then get the client to reconnect.


topicConnection.setExceptionListener( new ExceptionListener() {
public void onException( JMSException e ) {
logger.logError( sessionLog( request, "Connection failed: " + e.getMessage() ) );
topicConnectionStatus.setOk( false );
}
});

// Continously pass 'retrieved' responses back to the client.
while ( topicConnectionStatus.isOk() ) {
// Receive response from ResponseStream topic.
// The receive method is called with timeout so we may check from time to time if the
// connection listener detected a connection failure.
ObjectMessage message = (ObjectMessage)topicSubscriber.receive( topicReveiveTimeout );

... do stuff
}

} finally {

if ( topicConnection != null ) {
try {
topicConnection.close();
...

Hope this helps

[alanc_yang]

In the case of receive timeout, the connection is closed. In order to receive the message again, the connection/session/subsdriber all need to be recreated? So if there is no message available on the tipic destination, every timeout interval the connection will be destroyed and rebuild? What's the reasonable timeout interval, min/hour/day?

Thanks,

[adrian.brock]

? A receive timeout does not cause the connection to be closed.

It is better for onException() to do the reconnect by closing the connection
and reconnectioning - see the version in JBoss's MDB container.
http://cvs.sourceforge.net/viewcvs.py/jboss/jboss/src/main/org/jboss/ejb/plugins/jms/JMSContainerInvoker.java?annotate=1.50.2.13

Regards,
Adrian

[skendorski]

I've just started occaisionally experiencing this at a customer site at about midnight. Thanks for the fix. My question is what are we fixing? Why does the connection fail? I've got heartbeats going back and forth every 30secs. Shouldn't this keep it alive? Or is there some other failure cause?

[yxyang]

Hi,

I have the same problem.
In my case, a standalone java application listens on a queue, After one days working, this problem happened. I have an onException implemented in the application and also, the onException() is called. Within this onException(), i try to close() the oldconnection first, and then try to reestablish a connection. But it seems that the close() of the old connection never returns. So, there is no chance to reconnect. So, i decide to reconnect first and then use another thread to close the old connection now(Not test yet, and don't know whether it solve the problem or not).

In addition, i am wondering what happend to the connection. In my case , both listner and the jboss server are running in the same machine.

public synchronized void onException(JMSException jmsException) {
log.error("onException() is called at " + new Date());
System.out.println("onException() is called at " + new Date());
jmsException.printStackTrace();
oldconnection = queueConnection;
for (int i = 0; i < 10; i++) {
try {
ConnectorMessageListener.this.init();
} catch (Exception e2) {
log.error("reconnection failed");
System.out.println("reconnection failed");
try {
log.error("sleep for 2 seconds and try reconnect again");
Thread.sleep(2000);
} catch (InterruptedException e1) {
e1.printStackTrace();
}
}
}

Thread reconnThread = new Thread() {
public void run() {
for (int i = 0; i < 100; i++) {
try {
if (oldconnection != null) {
oldconnection.close();
oldconnection = null;
}
return;
} catch (Exception e) {

log.error("disconnect the old connection failed");
try {
log.error("sleep for 2 seconds and try disconnect again");
Thread.sleep(2000);
} catch (InterruptedException e1) {
e1.printStackTrace();
}
}
}
log.debug("disconnect the old connection success");
System.out.println("disconnect the old connection success");
}
};
reconnThread.start();
}



regards

yang

[adrian.brock]

I'll bet if you debug it, you'll see something like:

thread1 - exceptionListener.onException()
thread1 - thread2.start();

thread2 - connection.close()
thread2 - exceptionListener.onException()
thread2 - thread3.start()

etc.

Try doing oldConnection.setExceptionListener(null) before closing so that when
close tries to invoke the server it doesn't end up back in onException()

I've been thinking about putting this at the start of Connection.close(), but I haven't
convinced myself that this doesn't lead to a deadlock.