1 2 Previous Next 15 Replies Latest reply on Sep 30, 2003 4:07 AM by vincentyeh

    Ping time out

    vincentyeh Newbie

      I wanted to use JBoss on one online system. The program worked. But JBoss server outputted the following errors after about 2 hours.
      I used JBoss 3.2.1 on Solaris 8, and I used JBossMQ and MessageDriven Beans.
      =====================================
      2003-09-23 17:28:04,919 WARN [org.jboss.mq.Connection] Connection failure:
      org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping time
      out.)
      at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
      at java.lang.Thread.run(Thread.java:536)
      Caused by: java.io.IOException: ping timeout.
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
      ... 2 more
      2003-09-23 17:28:04,921 WARN [org.jboss.mq.Connection] Connection failure:
      org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping time
      out.)
      at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
      at java.lang.Thread.run(Thread.java:536)
      Caused by: java.io.IOException: ping timeout.
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
      ... 2 more
      ======================================

      Could anybody help me to resolve this problem?
      Thanks in advance.

      Best regards,

      Vincent Yeh

        • 1. Re: Ping time out
          Stephane Nicoll Master

          What Invoker are you using? HTTP?

          Regards,

          Stephane

          • 2. Re: Ping time out
            vincentyeh Newbie

            Yes, the program process is as following:
            Client => Servlet (in JBoss) => JBossMQ (in JBoss)=> MessageDriven Beans (in JBoss)

            Best regards,

            Vincent Yeh

            • 3. Re: Ping time out
              Stephane Nicoll Master

              are you caching the jms session in some way? For performance reason it has a sense but this should be a problem after some time.

              Regards,

              Stephane

              • 4. Re: Ping time out
                Adrian Brock Master

                What error do you get on the server?
                Do you have enough threads configured for tomcat/jetty
                to handle the requests?

                Regards,
                Adrian

                • 5. Re: Ping time out
                  vincentyeh Newbie

                  Hi, Stephane:

                  Thanks for your response :)

                  I create JMS session in servlet, and I don't cache any session.
                  ======================================
                  InitialContext ctx = new InitialContext();
                  QueueConnectionFactory qcf = (QueueConnectionFactory) ctx.lookup("ConnectionFactory");
                  QueueConnection qConn = qcf.createQueueConnection();
                  QueueSession qSession = qConn.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);

                  Queue queue = (Queue) ctx.lookup("queue/B");

                  QueueSender qSender = qSession.createSender(queue);

                  MapMessage mapMsg = qSession.createMapMessage();
                  mapMsg.setLong("SendTime", System.currentTimeMillis());
                  .....
                  Queue queA = (Queue) ctx.lookup("queue/A");
                  mapMsg.setJMSReplyTo(queA);
                  .....
                  qSender.send(mapMsg);
                  ======================================
                  Best regards,

                  Vincent Yeh

                  • 6. Re: Ping time out
                    vincentyeh Newbie

                    Hi, Adrian:

                    Thanks for your response :)

                    After server outputted the messages as my first post messages, the MDB can't receive any message from JBossMQ.

                    I guest that you are maybe right :), because there are many concurrent requests sometimes.

                    But I don't know where to configure the thread number in Jetty/Tomcat.

                    Could you tell me how to configure it?

                    Thanks a lot,

                    Vincent Yeh

                    • 7. Re: Ping time out
                      Adrian Brock Master

                      server/default/deploy/jbossweb-xxx.sar/META-INF/jboss-service.xml

                      xxx == tomcat41 or jetty

                      Regards,
                      Adrian

                      • 8. Re: Ping time out
                        vincentyeh Newbie

                        Hi, Adrian:

                        Thanks for your response :)

                        I have updated the $JBOSS_HOME/server/default/deploy/jbossweb-jetty.sar/META-INF/jboss-service.xml, and
                        I seted
                        30
                        1500

                        But it didn't work.

                        Can I set the connection pool size to JBossMQ?

                        Best regards,

                        Vincent Yeh

                        • 9. Re: Ping time out
                          Adrian Brock Master

                          Then threads is not the problem.

                          There is no such thing as a connection pool for jms.

                          Back to my original question, what error do you get
                          on the server?

                          Regards,
                          Adrian

                          • 10. Re: Ping time out
                            vincentyeh Newbie

                            Hi, Adrian:

                            I used JMeter 1.9.1 to test program, and I set 200 threads in 1 second with 10 loop count.
                            The server outputted the following messages:
                            ======================================
                            2003-09-26 09:47:21,614 DEBUG [org.jboss.mq.Connection] Error closing the connection
                            java.lang.NullPointerException
                            at org.jboss.mq.il.oil.OILServerIL.destroyConnection(OILServerIL.java:586)
                            at org.jboss.mq.il.oil.OILServerIL.connectionClosing(OILServerIL.java:338)
                            at org.jboss.mq.Connection.(Connection.java:256)
                            at org.jboss.mq.Connection.(Connection.java:315)
                            at org.jboss.mq.SpyConnection.(SpyConnection.java:59)
                            at org.jboss.mq.SpyConnectionFactory.createQueueConnection(SpyConnectionFactory.java:116)
                            at tw.com.elitec.smg.SMGServlet.doGet(Unknown Source)
                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
                            at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:360)
                            at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294)
                            at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558)
                            at org.mortbay.http.HttpContext.handle(HttpContext.java:1714)
                            at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:507)
                            at org.mortbay.http.HttpContext.handle(HttpContext.java:1664)
                            at org.mortbay.http.HttpServer.service(HttpServer.java:863)
                            at org.jboss.jetty.Jetty.service(Jetty.java:460)
                            at org.mortbay.http.HttpConnection.service(HttpConnection.java:775)
                            at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:939)
                            at org.mortbay.http.HttpConnection.handle(HttpConnection.java:792)
                            at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:201)
                            at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289)
                            at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:455)
                            2003-09-26 09:47:21,631 ERROR [tw.com.elitec.smg.SMGServlet] SMGServlet.doGet() JMSException
                            org.jboss.mq.SpyJMSException: Cannot authenticate user; - nested throwable: (java.net.ConnectExcepti
                            on: Connection refused)
                            at org.jboss.mq.Connection.authenticate(Connection.java:883)
                            at org.jboss.mq.Connection.(Connection.java:238)
                            at org.jboss.mq.Connection.(Connection.java:315)
                            at org.jboss.mq.SpyConnection.(SpyConnection.java:59)
                            at org.jboss.mq.SpyConnectionFactory.createQueueConnection(SpyConnectionFactory.java:116)
                            at tw.com.elitec.smg.SMGServlet.doGet(Unknown Source)
                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
                            at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:360)
                            at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294)
                            at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558)
                            at org.mortbay.http.HttpContext.handle(HttpContext.java:1714)
                            at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:507)
                            at org.mortbay.http.HttpContext.handle(HttpContext.java:1664)
                            at org.mortbay.http.HttpServer.service(HttpServer.java:863)
                            at org.jboss.jetty.Jetty.service(Jetty.java:460)
                            at org.mortbay.http.HttpConnection.service(HttpConnection.java:775)
                            at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:939)
                            at org.mortbay.http.HttpConnection.handle(HttpConnection.java:792)
                            at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:201)
                            at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289)
                            at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:455)
                            Caused by: java.net.ConnectException: Connection refused
                            at java.net.PlainSocketImpl.socketConnect(Native Method)
                            at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:305)
                            at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:171)
                            at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:158)
                            at java.net.Socket.connect(Socket.java:426)
                            at java.net.Socket.connect(Socket.java:376)
                            at java.net.Socket.(Socket.java:291)
                            at java.net.Socket.(Socket.java:147)
                            at javax.net.DefaultSocketFactory.createSocket(DashoA6275)
                            at org.jboss.mq.il.oil.OILServerIL.createConnection(OILServerIL.java:563)
                            at org.jboss.mq.il.oil.OILServerIL.checkConnection(OILServerIL.java:507)
                            at org.jboss.mq.il.oil.OILServerIL.authenticate(OILServerIL.java:289)
                            at org.jboss.mq.Connection.authenticate(Connection.java:876)
                            ... 21 more
                            ======================================
                            I set
                            60
                            2000

                            In My Servlet doGet() method, the catch-finally block is following:
                            ======================================
                            .....
                            } catch (IOException e) {
                            request.setAttribute("javax.servlet.jsp.jspException", e);
                            log.error("SMGServlet.doGet() IOException", e);
                            } catch (JMSException e) {
                            log.error("SMGServlet.doGet() JMSException", e);
                            } catch (Exception e) {
                            log.error("SMGServlet.doGet() Exception", e);
                            } finally {
                            try {
                            if (qSender != null) { qSender.close(); }
                            if (qSession != null) { qSession.close(); }
                            if (qConn != null) { qConn.close(); }
                            } catch (Exception e) {
                            log.error("SMGServlet.doGet() Close JMS Connection Exception", e);
                            }
                            }

                            ======================================

                            Best regards,

                            Vincent Yeh

                            • 11. Re: Ping time out
                              Adrian Brock Master

                              Looks like the server cannot cope with 200 connections
                              in one second?
                              We don't let you set the backlog, but the default is 200
                              so I don't see why it should fail, must be at the OS level.

                              Is the servlet on the same server as JMS?
                              You can avoid the network altogether if you lookup
                              java:/ConnectionFactory.

                              Regards,
                              Adrian

                              • 12. Re: Ping time out
                                vincentyeh Newbie

                                Hi, Adrian:

                                Thanks for your responses and efforts :)

                                The servlet is also in the same server, how do I check OS level? ulimit -a?
                                =====================
                                time(seconds) unlimited
                                file(blocks) unlimited
                                data(kbytes) unlimited
                                stack(kbytes) 8192
                                coredump(blocks) unlimited
                                nofiles(descriptors) 256
                                vmemory(kbytes) unlimited
                                =====================

                                Sometimes there are over 200 concurrent requests in the online system, whether I should change MDB to Session Beans or not?

                                Best regards,

                                Vincent Yeh

                                • 13. Re: Ping time out
                                  lhankins Newbie

                                  I've seen a similar issue using JBoss 3.2.1 and JDK 1.4.2_01 on WinXP.

                                  Its very sporadic and hard to reproduce, but is definitely NOT load based.

                                  Basically, we start getting these at some point on the server (repeating about every one minute)



                                  ----------------------------------------------------------------
                                  (Server Side Output)


                                  22:42:09,292 WARN [Connection]: Connection failure:
                                  org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
                                  at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
                                  at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
                                  at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
                                  at java.lang.Thread.run(Thread.java:534)
                                  Caused by: java.io.IOException: ping timeout.
                                  at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
                                  ... 2 more

                                  ----------------------------------------------------------------


                                  I think I've narrowed it down to our "ModelCache" class. ModelCache is a server side singleton which also happens to be a JMS listener. Here's another error message that gets generated when we get into this state:


                                  ----------------------------------------------------------------
                                  (Server Side Output)


                                  22:51:39,039 ERROR [ModelCache]: exception occurred [Cannot send a message to the JMS server; - nested throwable: (java.io.IOException: Client is not connected)]
                                  org.jboss.mq.SpyJMSException: Cannot send a message to the JMS server; - nested throwable: (java.io.IOException: Client is not connected)
                                  at org.jboss.mq.Connection.sendToServer(Connection.java:1251)
                                  at org.jboss.mq.SpySession.sendMessage(SpySession.java:699)
                                  at org.jboss.mq.SpyTopicPublisher.internalPublish(SpyTopicPublisher.java:120)
                                  at org.jboss.mq.SpyTopicPublisher.publish(SpyTopicPublisher.java:68)
                                  at com.interfacing.models.services.ModelCache.publishMessage(ModelCache.java:338)
                                  at com.interfacing.models.services.ModelCache.vbosWereCheckedOut(ModelCache.java:271)
                                  at com.interfacing.models.services.ModelRepositoryEjb.checkoutVbos(ModelRepositoryEjb.java:413)
                                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                  at java.lang.reflect.Method.invoke(Method.java:324)
                                  at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:629)
                                  at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
                                  at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:72)
                                  at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
                                  at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:273)
                                  at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:104)
                                  at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:117)
                                  at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191)
                                  at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
                                  at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java:322)
                                  at org.jboss.ejb.Container.invoke(Container.java:674)
                                  at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
                                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                  at java.lang.reflect.Method.invoke(Method.java:324)
                                  at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
                                  at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549)
                                  at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:359)
                                  at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
                                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                  at java.lang.reflect.Method.invoke(Method.java:324)
                                  at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261)
                                  at sun.rmi.transport.Transport$1.run(Transport.java:148)
                                  at java.security.AccessController.doPrivileged(Native Method)
                                  at sun.rmi.transport.Transport.serviceCall(Transport.java:144)
                                  at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
                                  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
                                  at java.lang.Thread.run(Thread.java:534)
                                  Caused by: java.io.IOException: Client is not connected
                                  at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:225)
                                  at org.jboss.mq.il.uil2.SocketManager.sendMessage(SocketManager.java:189)
                                  at org.jboss.mq.il.uil2.UILServerIL.addMessage(UILServerIL.java:235)
                                  at org.jboss.mq.Connection.sendToServer(Connection.java:1243)
                                  ... 37 more

                                  -------------------------------------------


                                  One of our customers has reported this bug in their production environment, but its been very hard for me to reproduce in my development environment (I did finally manage to reproduce it tonight).

                                  One hypothesis I had was that this type of thing could occur during hot-deploy... since the ModelCache is a singleton, things get a little strange during a hot-deploy (newly deployed ear = new classloader = new copy of singleton).

                                  Anyway - the fact that it has happened in the production environment kinda harpoons that theory though (no hot-deploys occur in production).

                                  I did try upgrading to the UIL2 invocation layer recently in hopes it might fix this, but apparently that didn't work...

                                  • 14. Re: Ping time out
                                    Adrian Brock Master

                                    Does your machine suffer from paging memory <-> disk?

                                    This can break the garbage collector.

                                    I have seen it where a gc took more than one minute
                                    because of paging which caused the jms ping to fail
                                    and the connection was closed.

                                    You can check for a failure by installing an ExceptionListener
                                    on the connection that re-establishes the connection.

                                    If you are in the same VM as the JMS server you can use
                                    the java:/ConnectionFactory which is just method calls and
                                    therefore doesn't require a ping to check the connection
                                    is still valid.

                                    Regards,
                                    Adrian

                                    1 2 Previous Next