12 Replies Latest reply on Aug 26, 2008 12:41 PM by Joe Mihalich

    Slight problem still regarding the fix for bug JBAS-1489

    Joe Mihalich Newbie


      Hi,

      We were experiencing the issue described in bug 1489. We successfully patched our system with the files that were fixed for this bug, patching just jbossmq.jar.

      In our case, the thing that caused the JMS server's socket to die was someone pulling the network cable from the network card on our box. Now, when the cable gets plugged back in, everything reconnects as designed with the fix.

      Howevere, there remains one issue. Even though the connections are re-established, we still get this exception repeatitively:

      2005-08-05 15:01:30,559 [hread] WARN org.jboss.mq.Connection : Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error and reconnect
      org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1307)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
      at java.lang.Thread.run(Thread.java:534)
      Caused by: java.io.IOException: ping timeout.
      ... 3 more
      2005-08-05 15:01:30,700 [hread] WARN org.jboss.mq.Connection : Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error and reconnect
      org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1307)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
      at java.lang.Thread.run(Thread.java:534)
      Caused by: java.io.IOException: ping timeout.
      ... 3 more


      Prior to those messages continuing, I see this message for all topics that are running on our server:

      2005-08-05 14:48:56,655 [hread] INFO org.jboss.ejb.plugins.jms.JMSContainerInvoker : Reconnected to JMS provider


      Is this a bug that is still lingering? It's not harming anything that I can notice, but this should go away once all connections are re-established right?


      Thanks,
      Joe

        • 1. Re: Slight problem still regarding the fix for bug JBAS-1489
          Adrian Brock Master

           

          "jmihalich" wrote:

          2005-08-05 14:48:56,655 [hread] INFO org.jboss.ejb.plugins.jms.JMSContainerInvoker : Reconnected to JMS provider


          JBAS-1489 is completely unrelated to MDBs. It is for user created MessageListeners.

          Useful links:
          http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossHelp
          and
          READ THIS FIRST

          2005-08-05 15:01:30,559 [hread] WARN org.jboss.mq.Connection : Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error and reconnect
          

          It is almost worth adding a Stacktrace of where the connection was allocated
          to this error message, then I wouldn't have to ask for the TRACE logging all the time
          (not that I do anymore, I just ignore people that cannot READ THIS FIRST).

          P.S. I'm only interested in this problem if you can show the connection that is still
          trying to ping is from the MDB or java:/JmsXA
          AND it is still a problem in the latest release.

          • 2. Re: Slight problem still regarding the fix for bug JBAS-1489
            shadkam Newbie

            Hi,

            Even we are facing the same problem mentioned by Joe. But for us, the same code works fine on BSD and Linux, but does not work on windows XP.
            Following is the scenario:
            o Server is up, happily posting msgs to the clients, everything is fine.
            o Somebody pulls out the network cable of the server machine
            o Understandably, it is no longer able to send any msgs, and it shows Exceptions pasted below.
            o Now, even when the network cable is connected back, other things (like request response / session EJB calls) work fine, but the JMS never seems to recover. It keeps on printing the same Exceptions for every call to publish method.

            19:50:00,682 WARN [Connection] Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error
            and reconnect
            org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable: (java.io.IOException: Client is not connected)
            at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
            at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
            at org.jboss.mq.Connection.pingServer(Connection.java:887)
            at org.jboss.mq.Connection$PingTask.run(Connection.java:1327)
            at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
            at java.lang.Thread.run(Thread.java:595)
            Caused by: java.io.IOException: Client is not connected
            at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
            at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:224)
            at org.jboss.mq.il.uil2.UILServerIL.ping(UILServerIL.java:462)
            at org.jboss.mq.Connection.pingServer(Connection.java:883)
            ... 3 more
            19:50:00,698 WARN [Connection] Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error
            and reconnect
            org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable: (java.io.IOException: Client is not connec
            ted)
            at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
            at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
            at org.jboss.mq.Connection.pingServer(Connection.java:887)
            at org.jboss.mq.Connection$PingTask.run(Connection.java:1327)
            at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
            at java.lang.Thread.run(Thread.java:595)
            Caused by: java.io.IOException: Client is not connected
            at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
            at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:224)
            at org.jboss.mq.il.uil2.UILServerIL.ping(UILServerIL.java:462)
            at org.jboss.mq.Connection.pingServer(Connection.java:883)
            ... 3 more
            19:50:00,745 WARN [Connection] Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error
            and reconnect
            org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable: (java.io.IOException: Client is not connected)
            at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
            at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
            at org.jboss.mq.Connection.pingServer(Connection.java:887)
            at org.jboss.mq.Connection$PingTask.run(Connection.java:1327)
            at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
            at java.lang.Thread.run(Thread.java:595)
            Caused by: java.io.IOException: Client is not connected
            at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
            at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:224)
            at org.jboss.mq.il.uil2.UILServerIL.ping(UILServerIL.java:462)
            at org.jboss.mq.Connection.pingServer(Connection.java:883)
            ... 3 more
            19:50:00,761 WARN [Connection] Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error
            and reconnect
            org.jboss.mq.SpyJMSException: Cannot ping the JMS server; - nested throwable: (java.io.IOException: Client is not connected)
            at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
            at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
            at org.jboss.mq.Connection.pingServer(Connection.java:887)
            at org.jboss.mq.Connection$PingTask.run(Connection.java:1327)
            at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
            at java.lang.Thread.run(Thread.java:595)
            Caused by: java.io.IOException: Client is not connected
            at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
            at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:224)
            at org.jboss.mq.il.uil2.UILServerIL.ping(UILServerIL.java:462)
            at org.jboss.mq.Connection.pingServer(Connection.java:883)
            ... 3 more

            • 3. Re: Slight problem still regarding the fix for bug JBAS-1489
              shadkam Newbie

              Also, I forgot to mention one thing ... we did try setting exceptionListener and then handle the exception, we re-got the ConnectionFactory from JNDI and created the sessions etc afresh ... but that did not help, perhaps because the ConnectionFactory itself had become corrupted (its internal sockets) ... apart from redeployment, is there a way to correct those ...
              thanks
              shadkam

              • 4. Re: Slight problem still regarding the fix for bug JBAS-1489
                shadkam Newbie

                Also, I forgot to mention one thing ... we did try setting exceptionListener and then handle the exception, we re-got the ConnectionFactory from JNDI and created the sessions etc afresh ... but that did not help, perhaps because the ConnectionFactory itself had become corrupted (its internal sockets) ... apart from redeployment, is there a way to correct those ...
                thanks
                shadkam

                • 5. Re: Slight problem still regarding the fix for bug JBAS-1489
                  Johan Kumps Newbie

                  Any thought on this? I'm currently facing the exact same problem. (Windows XP)

                  • 6. Re: Slight problem still regarding the fix for bug JBAS-1489
                    uday kumar Newbie

                    I'm facing the same problem. I don't know what went wrong in that. Please help me in this context. It's Urjent Please......Please....Please....

                    • 7. Re: Slight problem still regarding the fix for bug JBAS-1489
                      Adrian Brock Master

                      Which part of


                      use javax.jms.Connection.setExceptionListener() to handle this error
                      and reconnect

                      don't you guys understaand?

                      • 8. Re: Slight problem still regarding the fix for bug JBAS-1489
                        Mark Gonsalves Newbie

                        Well Adrian, that is exactly what I did. When I initialize queue, I register for exception callbacks...

                        conn = qcf.createQueueConnection();
                        // Register for exception callbacks
                        conn.setExceptionListener(new ExceptionListenerImpl());
                        session = conn.createQueueSession(false, QueueSession.AUTO_ACKNOWLEDGE);
                        


                        I got the idea from here...

                        http://wiki.jboss.org/wiki/JMSConnectionExceptionListener

                        Looks pretty straight forward. So after implementing the exception listener and redeploying, I still get the failure below.

                        10:42:08,781 WARN [Connection] Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error
                         and reconnect
                        org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
                         at org.jboss.mq.Connection$PingTask.run(Connection.java:1313)
                         at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
                         at java.lang.Thread.run(Thread.java:619)
                        Caused by: java.io.IOException: ping timeout.
                         ... 3 more
                        


                        Interesting this is that I have added logging to the callback implementation and none of it shows up on the console. This tells me that the call back isn't being called.

                        Am I doing something wrong? It all started when I lost network connectivity for a few seconds. Also, my jboss instance is bound to the ip address of the machine it is running on, instead of the localhost default.

                        • 9. Re: Slight problem still regarding the fix for bug JBAS-1489
                          Adrian Brock Master

                           

                          "beefcaaaake" wrote:
                          Well Adrian, that is exactly what I did.


                          No you didn't, otherwise you wouldn't get that error message.

                          Try
                          conn = qcf.createQueueConnection();
                          // Register for exception callbacks
                          conn.setExceptionListener(new ExceptionListenerImpl());
                          System.out.println("=========> Now running the new code");
                          session = conn.createQueueSession(false, QueueSession.AUTO_ACKNOWLEDGE);
                          


                          There's also an FAQ on how to track down where you forgot to do it.

                          Usually when this message comes up, people swear blind they've done
                          it correctly and then find they're creating a connection somewhere else
                          which doesn't have the exception listener. :-)

                          • 10. Re: Slight problem still regarding the fix for bug JBAS-1489
                            kevin m Newbie

                            We are experiencing a very similar problem, and I was hoping on clarification on whether or not there is a bug here, and if it has been fixed. Or if there is a way to debug this issue.

                            The server is 4.0.3SP1 jboss server on a linux box. The client is 4.0.2.

                            Our JMS client has a topic connection that has a listener. The listener sets a flag, and another thread will [very quickly] clean up and then reconnect when it sees the flag set.

                            The problems began when there was a network outage.

                            Since then, we've been in a cycling loop of reconnects. The client gets a "no pong received". The onException is invoked. The thread cleans up and reconnects. Messages are transferred from the client to the server, and from the server to the client. A minute later, we get a "no pong received" exception - repeat.

                            2008-08-06 09:46:01,416 WARN [TopicConnectionManager] the connection to the management server was broken: org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
                            2008-08-06 09:46:02,112 WARN [TopicConnectionManager] successfully established a connection to the management server
                            2008-08-06 09:47:01,416 WARN [TopicConnectionManager] the connection to the management server was broken: org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
                            2008-08-06 09:47:02,459 WARN [TopicConnectionManager] successfully established a connection to the management server
                            2008-08-06 09:48:01,420 WARN [TopicConnectionManager] the connection to the management server was broken: org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
                            2008-08-06 09:48:01,811 WARN [TopicConnectionManager] successfully established a connection to the management server
                            2008-08-06 09:49:01,424 WARN [TopicConnectionManager] the connection to the management server was broken: org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
                            2008-08-06 09:49:02,152 WARN [TopicConnectionManager] successfully established a connection to the management server
                            2008-08-06 09:50:01,424 WARN [TopicConnectionManager] the connection to the management server was broken: org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
                            2008-08-06 09:50:01,500 WARN [TopicConnectionManager] successfully established a connection to the management server
                            2008-08-06 09:51:01,428 WARN [TopicConnectionManager] the connection to the management server was broken: org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)


                            Here's what the debug log shows.

                            2008-08-06 10:20:01,476 DEBUG [TopicConnectionManager] exception reported on topic connection
                            org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.)
                             at org.jboss.mq.Connection$PingTask.run(Connection.java:1323)
                             at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
                             at java.lang.Thread.run(Thread.java:619)
                            Caused by: java.io.IOException: ping timeout.
                             ... 3 more


                            We have seen the same exact problem on many clients on different machines, connected to the same server. The log file shows a "no pong received" exception delivered every 60 seconds, apparently forever.

                            There are no huge messages, or large numbers of exceptions. This is verified with tcpdump.

                            Since JMS seems like its otherwise working - server sees what the client published, client sees what the server published - this looks like some sort of JBoss issue.

                            Can anyone help? I'm not sure what else it would be helpful to know.

                            • 11. Re: Slight problem still regarding the fix for bug JBAS-1489
                              Cory Dahlstrom Newbie

                              We are experiencing this problem as well. The result of this issue is we eventually receive a "too many open files" error. I assume this is the result of the "no pong received" which prompts the server to create a new connection. We can see the established sockets ( 100s of them ) before this error occurs using "netstat -an".

                              Our environment is JBoss 4.2.0GA. We have a JBoss server running which contains our JMS queues and one that connects remotely to those queues. If the server which contains the queues is restarted, we randomly get the ping error. It doesn't happen every time. I assume there is a timing issue here as to what state the server is in that has the queues and when the remote server tries to ping it.

                              In looking at the org.jboss.mq.Connection code, when this situation occurs, the "PingTask" runnable never gets to the "pingServer()" call because the "ponged" variable is always false. If the ponged variable is false, an exception is thrown causing it to never get to the "pingServer()".

                              Has this been fixed in a latter release of JBoss?

                              Thanks,

                              Cory.

                              • 12. Re: Slight problem still regarding the fix for bug JBAS-1489
                                Joe Mihalich Newbie

                                I gave up trying to fix this problem. It didn't seem to be affecting anything, as you said. It just appears there's a thread out there that is still running that should have exited.

                                We switched to jboss messaging about 4 months ago, on jboss 4.2.2. All the problems we had with jboss mq with jboss mq deadlocking on heavy load, and this pong stuff went away. The new db schema and code they're using is so much better and more stable.

                                Joe