1 2 Previous Next 20 Replies Latest reply on Apr 26, 2004 9:35 AM by Dusan Hlavac

    JMS Ping Timeout

    lucasfowler Newbie

      We are getting the following connection failed message whilst listening
      on a topic.
      The topic itself is ok, because if we connect a new client then it can read without
      problems.

      [2004-03-12 12:03:36,273] [WARN ] [org.jboss.mq.Connection] Connection failure:
      org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
      at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Unknown Source)
      at java.lang.Thread.run(Thread.java:536)
      Caused by: java.io.IOException: ping timeout.
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
      ... 2 more

      I can provide a stack trace if required, but it is too big to post here

      Here is the stdout.log:
      ================================================================================

      JBoss Bootstrap Environment

      JBOSS_HOME: /home/trdmdmo/AppServer/trdmdmo/JBossTomcat

      JAVA: /opt2/JavaCentral/linux/j2sdk1.4.1_01/bin/java

      JAVA_OPTS: -DJBOSS=602 -server -server -Xms4m -Xmx128m -Dprogram.name=run.sh

      CLASSPATH: /home/trdmdmo/AppServer/trdmdmo/JBossTomcat/bin/run.jar:/opt2/JavaCentral/linux/j2sdk1.4.1_01/lib/tools.jar

      ================================================================================

      06:08:33,140 INFO [Server] Starting JBoss (MX MicroKernel)...
      06:08:33,144 INFO [Server] Release ID: JBoss [WonderLand] 3.2.3 (build: CVSTag=JBoss_3_2_3 date=200311301445)
      06:08:33,154 INFO [Server] Home Dir: /home/trdmdmo/AppServer/trdmdmo/JBossTomcat
      06:08:33,154 INFO [Server] Home URL: file:/home/trdmdmo/AppServer/trdmdmo/JBossTomcat/
      06:08:33,155 INFO [Server] Library URL: file:/home/trdmdmo/AppServer/trdmdmo/JBossTomcat/lib/
      06:08:33,158 INFO [Server] Patch URL: null
      06:08:33,158 INFO [Server] Server Name: default
      06:08:33,158 INFO [Server] Server Home Dir: /home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default
      06:08:33,159 INFO [Server] Server Home URL: file:/home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default/
      06:08:33,159 INFO [Server] Server Data Dir: /home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default/data
      06:08:33,160 INFO [Server] Server Temp Dir: /home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default/tmp
      06:08:33,160 INFO [Server] Server Config URL: file:/home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default/conf/
      06:08:33,168 INFO [Server] Server Library URL: file:/home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default/lib/
      06:08:33,168 INFO [Server] Root Deployment Filename: jboss-service.xml
      06:08:33,193 INFO [Server] Starting General Purpose Architecture (GPA)...
      06:08:33,899 INFO [ServerInfo] Java version: 1.4.1_01,Sun Microsystems Inc.
      06:08:33,899 INFO [ServerInfo] Java VM: Java HotSpot(TM) Server VM 1.4.1_01-b01,Sun Microsystems Inc.
      06:08:33,899 INFO [ServerInfo] OS-System: Linux 2.4.18-686-smp,i386
      06:08:33,991 INFO [ServiceController] Controller MBean online
      06:08:34,180 INFO [MainDeployer] Started jboss.system:service=MainDeployer
      06:08:34,367 INFO [MainDeployer] Adding deployer: org.jboss.deployment.JARDeployer@1b09468
      06:08:34,368 INFO [JARDeployer] Started jboss.system:service=JARDeployer
      06:08:34,427 INFO [MainDeployer] Adding deployer: org.jboss.deployment.SARDeployer@861f24
      06:08:34,453 INFO [SARDeployer] Started jboss.system:service=ServiceDeployer
      06:08:34,454 INFO [Server] Core system initialized
      06:08:34,508 INFO [MainDeployer] Starting deployment of package: file:/home/trdmdmo/AppServer/trdmdmo/JBossTomcat/server/default/conf/jboss-service.xml

      Can anyone give us some pointers?

      Thanks

        • 1. Re: JMS Ping Timeout
          Adrian Brock Master

          javax.jms.Connection.setExceptionListener()

          Regards,
          Adrian

          • 2. Re: JMS Ping Timeout
            Martha L. Cofran Newbie

            I am currently experiencing the same issue that lucas has, but I don't understand what you're saying should be done. Are you suggesting we create an exception listener? What would it listen for and what would it do when it gets a message?

            • 3. Re: JMS Ping Timeout
              Christopher Rued Newbie

              I'm running into the same issue.
              In other threads, it looks like the suggestion was to re-initialize the connection when you receive an exception to try to recover.

              In my case, I have I have a temporary queue set up to receive a response from some message (that is, the reply-to field in some message I send is set to be my temporary queue). My understanding is that once a connection is closed, all temporary queues are destroyed -- how can I reinitialize the connection and have it receive messages on the temporary queue?

              The only solution I can come up with is to not use temporary queues, but I'm hoping to avoid this, if possible.

              Adrian? Anyone?

              Thanks in advance.

              • 4. Re: JMS Ping Timeout
                lucasfowler Newbie

                Thanks for the infos here.

                We have added an exceptionListener and that seems to have done the trick.

                • 5. Re: JMS Ping Timeout
                  alanc_yang Newbie

                  I encountered the same problem, but wasn't able to see the ExceptionListener's onException gets call when it happened. This was what I did:

                  1) in my MDBean to implement ExceptionListener
                  2) provide onException method
                  3) in the connection to setExceptionListener(this);

                  I must have missed something. Please help!
                  Thanks in advance.

                  Alan

                  • 6. Re: JMS Ping Timeout
                    Richard Bielak Newbie

                     

                    "lucasfowler" wrote:
                    Thanks for the infos here.

                    We have added an exceptionListener and that seems to have done the trick.


                    And what does your listener do?

                    ...richie

                    • 7. Re: JMS Ping Timeout
                      Andrew Tyson Newbie

                      Hi Guys,

                      I am also experiencing this 'problem' with 3.2.3 and a Queue, however the bug appears to be harmless. Can the ExceptionHandler just ignore the Exception, or should it perform some cleanup ?

                      Thanks and regards,
                      Andrew

                      • 8. Re: JMS Ping Timeout
                        alanc_yang Newbie

                        I have two subscribers, one MBean and one MDBean that in 1 min apart, the exceptoin was thrown as in the following:

                        07:57:05,473 WARN [Connection] Connection failure:
                        org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException:
                        ping timeout.)
                        at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
                        at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
                        at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
                        at java.lang.Thread.run(Thread.java:534)
                        Caused by: java.io.IOException: ping timeout.
                        at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
                        ... 2 more

                        Seems like the MDBean has no impact that it would continue receving the message. But the MBean failed. My MBean code was like this:

                        1. public class myMBean extends ServiceMBeanSupport ...
                        2. within startService(), it lanuches another thread
                        3. the other thread performs a while loop:
                        while (true) {
                        setupConnection();

                        msg = sub.receive();
                        processMsg(msg);

                        if(msg == null) System.out.println(" receiving alarm failed");
                        stopConnection();
                        }

                        The MBean would fail after the above exception. Would like to know is this the right way to code the message receiving loop, or how to work around the problem.

                        Thanks a lot.
                        Alan


                        • 9. Re: JMS Ping Timeout
                          lucasfowler Newbie

                           

                          "richieb" wrote:
                          "lucasfowler" wrote:
                          Thanks for the infos here.

                          We have added an exceptionListener and that seems to have done the trick.


                          And what does your listener do?

                          ...richie


                          OK, we are running in a servlet so we can't start another thread. So we
                          set up the listener to set a flag and changed the read method to timeout
                          every so often. That way, if there is a connection failure we can close
                          properly in our finally clause and then get the client to reconnect.


                          topicConnection.setExceptionListener( new ExceptionListener() {
                          public void onException( JMSException e ) {
                          logger.logError( sessionLog( request, "Connection failed: " + e.getMessage() ) );
                          topicConnectionStatus.setOk( false );
                          }
                          });

                          // Continously pass 'retrieved' responses back to the client.
                          while ( topicConnectionStatus.isOk() ) {
                          // Receive response from ResponseStream topic.
                          // The receive method is called with timeout so we may check from time to time if the
                          // connection listener detected a connection failure.
                          ObjectMessage message = (ObjectMessage)topicSubscriber.receive( topicReveiveTimeout );

                          ... do stuff
                          }

                          } finally {

                          if ( topicConnection != null ) {
                          try {
                          topicConnection.close();
                          ...

                          Hope this helps

                          • 10. Re: JMS Ping Timeout
                            alanc_yang Newbie

                            In the case of receive timeout, the connection is closed. In order to receive the message again, the connection/session/subsdriber all need to be recreated? So if there is no message available on the tipic destination, every timeout interval the connection will be destroyed and rebuild? What's the reasonable timeout interval, min/hour/day?

                            Thanks,

                            • 11. Re: JMS Ping Timeout
                              Adrian Brock Master

                              ? A receive timeout does not cause the connection to be closed.

                              It is better for onException() to do the reconnect by closing the connection
                              and reconnectioning - see the version in JBoss's MDB container.
                              http://cvs.sourceforge.net/viewcvs.py/jboss/jboss/src/main/org/jboss/ejb/plugins/jms/JMSContainerInvoker.java?annotate=1.50.2.13

                              Regards,
                              Adrian

                              • 12. Re: JMS Ping Timeout
                                Stephen Kendorski Newbie

                                I've just started occaisionally experiencing this at a customer site at about midnight. Thanks for the fix. My question is what are we fixing? Why does the connection fail? I've got heartbeats going back and forth every 30secs. Shouldn't this keep it alive? Or is there some other failure cause?

                                • 13. Re: JMS Ping Timeout
                                  peter Apprentice

                                  Hi,

                                  I have the same problem.
                                  In my case, a standalone java application listens on a queue, After one days working, this problem happened. I have an onException implemented in the application and also, the onException() is called. Within this onException(), i try to close() the oldconnection first, and then try to reestablish a connection. But it seems that the close() of the old connection never returns. So, there is no chance to reconnect. So, i decide to reconnect first and then use another thread to close the old connection now(Not test yet, and don't know whether it solve the problem or not).

                                  In addition, i am wondering what happend to the connection. In my case , both listner and the jboss server are running in the same machine.

                                  public synchronized void onException(JMSException jmsException) {
                                  log.error("onException() is called at " + new Date());
                                  System.out.println("onException() is called at " + new Date());
                                  jmsException.printStackTrace();
                                  oldconnection = queueConnection;
                                  for (int i = 0; i < 10; i++) {
                                  try {
                                  ConnectorMessageListener.this.init();
                                  } catch (Exception e2) {
                                  log.error("reconnection failed");
                                  System.out.println("reconnection failed");
                                  try {
                                  log.error("sleep for 2 seconds and try reconnect again");
                                  Thread.sleep(2000);
                                  } catch (InterruptedException e1) {
                                  e1.printStackTrace();
                                  }
                                  }
                                  }

                                  Thread reconnThread = new Thread() {
                                  public void run() {
                                  for (int i = 0; i < 100; i++) {
                                  try {
                                  if (oldconnection != null) {
                                  oldconnection.close();
                                  oldconnection = null;
                                  }
                                  return;
                                  } catch (Exception e) {

                                  log.error("disconnect the old connection failed");
                                  try {
                                  log.error("sleep for 2 seconds and try disconnect again");
                                  Thread.sleep(2000);
                                  } catch (InterruptedException e1) {
                                  e1.printStackTrace();
                                  }
                                  }
                                  }
                                  log.debug("disconnect the old connection success");
                                  System.out.println("disconnect the old connection success");
                                  }
                                  };
                                  reconnThread.start();
                                  }



                                  regards

                                  yang

                                  • 14. Re: JMS Ping Timeout
                                    Adrian Brock Master

                                    I'll bet if you debug it, you'll see something like:

                                    thread1 - exceptionListener.onException()
                                    thread1 - thread2.start();

                                    thread2 - connection.close()
                                    thread2 - exceptionListener.onException()
                                    thread2 - thread3.start()

                                    etc.

                                    Try doing oldConnection.setExceptionListener(null) before closing so that when
                                    close tries to invoke the server it doesn't end up back in onException()

                                    I've been thinking about putting this at the start of Connection.close(), but I haven't
                                    convinced myself that this doesn't lead to a deadlock.

                                    1 2 Previous Next