Ping time out

[vincentyeh]

I wanted to use JBoss on one online system. The program worked. But JBoss server outputted the following errors after about 2 hours.
I used JBoss 3.2.1 on Solaris 8, and I used JBossMQ and MessageDriven Beans.
=====================================
2003-09-23 17:28:04,919 WARN [org.jboss.mq.Connection] Connection failure:
org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping time
out.)
at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:536)
Caused by: java.io.IOException: ping timeout.
at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
... 2 more
2003-09-23 17:28:04,921 WARN [org.jboss.mq.Connection] Connection failure:
org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping time
out.)
at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:536)
Caused by: java.io.IOException: ping timeout.
at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
... 2 more
======================================

Could anybody help me to resolve this problem?
Thanks in advance.

Best regards,

Vincent Yeh

[stephanenicoll]

What Invoker are you using? HTTP?

Regards,

Stephane

[vincentyeh]

Yes, the program process is as following:
Client => Servlet (in JBoss) => JBossMQ (in JBoss)=> MessageDriven Beans (in JBoss)

Best regards,

Vincent Yeh

[stephanenicoll]

are you caching the jms session in some way? For performance reason it has a sense but this should be a problem after some time.

Regards,

Stephane

[adrian.brock]

What error do you get on the server?
Do you have enough threads configured for tomcat/jetty
to handle the requests?

Regards,
Adrian

[vincentyeh]

Hi, Stephane:

Thanks for your response :)

I create JMS session in servlet, and I don't cache any session.
======================================
InitialContext ctx = new InitialContext();
QueueConnectionFactory qcf = (QueueConnectionFactory) ctx.lookup("ConnectionFactory");
QueueConnection qConn = qcf.createQueueConnection();
QueueSession qSession = qConn.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);

Queue queue = (Queue) ctx.lookup("queue/B");

QueueSender qSender = qSession.createSender(queue);

MapMessage mapMsg = qSession.createMapMessage();
mapMsg.setLong("SendTime", System.currentTimeMillis());
.....
Queue queA = (Queue) ctx.lookup("queue/A");
mapMsg.setJMSReplyTo(queA);
.....
qSender.send(mapMsg);
======================================
Best regards,

Vincent Yeh

[vincentyeh]

Hi, Adrian:

Thanks for your response :)

After server outputted the messages as my first post messages, the MDB can't receive any message from JBossMQ.

I guest that you are maybe right :), because there are many concurrent requests sometimes.

But I don't know where to configure the thread number in Jetty/Tomcat.

Could you tell me how to configure it?

Thanks a lot,

Vincent Yeh

[adrian.brock]

server/default/deploy/jbossweb-xxx.sar/META-INF/jboss-service.xml

xxx == tomcat41 or jetty

Regards,
Adrian

[vincentyeh]

Hi, Adrian:

Thanks for your response :)

I have updated the $JBOSS_HOME/server/default/deploy/jbossweb-jetty.sar/META-INF/jboss-service.xml, and
I seted
30
1500

But it didn't work.

Can I set the connection pool size to JBossMQ?

Best regards,

Vincent Yeh

[adrian.brock]

Then threads is not the problem.

There is no such thing as a connection pool for jms.

Back to my original question, what error do you get
on the server?

Regards,
Adrian

[vincentyeh]

Hi, Adrian:

I used JMeter 1.9.1 to test program, and I set 200 threads in 1 second with 10 loop count.
The server outputted the following messages:
======================================
2003-09-26 09:47:21,614 DEBUG [org.jboss.mq.Connection] Error closing the connection
java.lang.NullPointerException
at org.jboss.mq.il.oil.OILServerIL.destroyConnection(OILServerIL.java:586)
at org.jboss.mq.il.oil.OILServerIL.connectionClosing(OILServerIL.java:338)
at org.jboss.mq.Connection.(Connection.java:256)
at org.jboss.mq.Connection.(Connection.java:315)
at org.jboss.mq.SpyConnection.(SpyConnection.java:59)
at org.jboss.mq.SpyConnectionFactory.createQueueConnection(SpyConnectionFactory.java:116)
at tw.com.elitec.smg.SMGServlet.doGet(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:360)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1714)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:507)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1664)
at org.mortbay.http.HttpServer.service(HttpServer.java:863)
at org.jboss.jetty.Jetty.service(Jetty.java:460)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:775)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:939)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:792)
at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:201)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:455)
2003-09-26 09:47:21,631 ERROR [tw.com.elitec.smg.SMGServlet] SMGServlet.doGet() JMSException
org.jboss.mq.SpyJMSException: Cannot authenticate user; - nested throwable: (java.net.ConnectExcepti
on: Connection refused)
at org.jboss.mq.Connection.authenticate(Connection.java:883)
at org.jboss.mq.Connection.(Connection.java:238)
at org.jboss.mq.Connection.(Connection.java:315)
at org.jboss.mq.SpyConnection.(SpyConnection.java:59)
at org.jboss.mq.SpyConnectionFactory.createQueueConnection(SpyConnectionFactory.java:116)
at tw.com.elitec.smg.SMGServlet.doGet(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:360)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1714)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:507)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1664)
at org.mortbay.http.HttpServer.service(HttpServer.java:863)
at org.jboss.jetty.Jetty.service(Jetty.java:460)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:775)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:939)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:792)
at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:201)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:455)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:305)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:171)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:158)
at java.net.Socket.connect(Socket.java:426)
at java.net.Socket.connect(Socket.java:376)
at java.net.Socket.(Socket.java:291)
at java.net.Socket.(Socket.java:147)
at javax.net.DefaultSocketFactory.createSocket(DashoA6275)
at org.jboss.mq.il.oil.OILServerIL.createConnection(OILServerIL.java:563)
at org.jboss.mq.il.oil.OILServerIL.checkConnection(OILServerIL.java:507)
at org.jboss.mq.il.oil.OILServerIL.authenticate(OILServerIL.java:289)
at org.jboss.mq.Connection.authenticate(Connection.java:876)
... 21 more
======================================
I set
60
2000

In My Servlet doGet() method, the catch-finally block is following:
======================================
.....
} catch (IOException e) {
request.setAttribute("javax.servlet.jsp.jspException", e);
log.error("SMGServlet.doGet() IOException", e);
} catch (JMSException e) {
log.error("SMGServlet.doGet() JMSException", e);
} catch (Exception e) {
log.error("SMGServlet.doGet() Exception", e);
} finally {
try {
if (qSender != null) { qSender.close(); }
if (qSession != null) { qSession.close(); }
if (qConn != null) { qConn.close(); }
} catch (Exception e) {
log.error("SMGServlet.doGet() Close JMS Connection Exception", e);
}
}

======================================

Best regards,

Vincent Yeh

[adrian.brock]

Looks like the server cannot cope with 200 connections
in one second?
We don't let you set the backlog, but the default is 200
so I don't see why it should fail, must be at the OS level.

Is the servlet on the same server as JMS?
You can avoid the network altogether if you lookup
java:/ConnectionFactory.

Regards,
Adrian

[vincentyeh]

Hi, Adrian:

Thanks for your responses and efforts :)

The servlet is also in the same server, how do I check OS level? ulimit -a?
=====================
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) unlimited
nofiles(descriptors) 256
vmemory(kbytes) unlimited
=====================

Sometimes there are over 200 concurrent requests in the online system, whether I should change MDB to Session Beans or not?

Best regards,

Vincent Yeh

[lhankins]

I've seen a similar issue using JBoss 3.2.1 and JDK 1.4.2_01 on WinXP.

Its very sporadic and hard to reproduce, but is definitely NOT load based.

Basically, we start getting these at some point on the server (repeating about every one minute)



----------------------------------------------------------------
(Server Side Output)


22:42:09,292 WARN [Connection]: Connection failure:
org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.io.IOException: ping timeout.
at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
... 2 more

----------------------------------------------------------------


I think I've narrowed it down to our "ModelCache" class. ModelCache is a server side singleton which also happens to be a JMS listener. Here's another error message that gets generated when we get into this state:


----------------------------------------------------------------
(Server Side Output)


22:51:39,039 ERROR [ModelCache]: exception occurred [Cannot send a message to the JMS server; - nested throwable: (java.io.IOException: Client is not connected)]
org.jboss.mq.SpyJMSException: Cannot send a message to the JMS server; - nested throwable: (java.io.IOException: Client is not connected)
at org.jboss.mq.Connection.sendToServer(Connection.java:1251)
at org.jboss.mq.SpySession.sendMessage(SpySession.java:699)
at org.jboss.mq.SpyTopicPublisher.internalPublish(SpyTopicPublisher.java:120)
at org.jboss.mq.SpyTopicPublisher.publish(SpyTopicPublisher.java:68)
at com.interfacing.models.services.ModelCache.publishMessage(ModelCache.java:338)
at com.interfacing.models.services.ModelCache.vbosWereCheckedOut(ModelCache.java:271)
at com.interfacing.models.services.ModelRepositoryEjb.checkoutVbos(ModelRepositoryEjb.java:413)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:629)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:72)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:273)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:104)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:117)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java:322)
at org.jboss.ejb.Container.invoke(Container.java:674)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549)
at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:359)
at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261)
at sun.rmi.transport.Transport$1.run(Transport.java:148)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:144)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:225)
at org.jboss.mq.il.uil2.SocketManager.sendMessage(SocketManager.java:189)
at org.jboss.mq.il.uil2.UILServerIL.addMessage(UILServerIL.java:235)
at org.jboss.mq.Connection.sendToServer(Connection.java:1243)
... 37 more

-------------------------------------------


One of our customers has reported this bug in their production environment, but its been very hard for me to reproduce in my development environment (I did finally manage to reproduce it tonight).

One hypothesis I had was that this type of thing could occur during hot-deploy... since the ModelCache is a singleton, things get a little strange during a hot-deploy (newly deployed ear = new classloader = new copy of singleton).

Anyway - the fact that it has happened in the production environment kinda harpoons that theory though (no hot-deploys occur in production).

I did try upgrading to the UIL2 invocation layer recently in hopes it might fix this, but apparently that didn't work...

[adrian.brock]

Does your machine suffer from paging memory <-> disk?

This can break the garbage collector.

I have seen it where a gc took more than one minute
because of paging which caused the jms ping to fail
and the connection was closed.

You can check for a failure by installing an ExceptionListener
on the connection that re-establishes the connection.

If you are in the same VM as the JMS server you can use
the java:/ConnectionFactory which is just method calls and
therefore doesn't require a ping to check the connection
is still valid.

Regards,
Adrian