-
1. Re: Ping time out
stephanenicoll Sep 23, 2003 4:48 AM (in response to vincentyeh)What Invoker are you using? HTTP?
Regards,
Stephane -
2. Re: Ping time out
vincentyeh Sep 23, 2003 7:39 AM (in response to vincentyeh)Yes, the program process is as following:
Client => Servlet (in JBoss) => JBossMQ (in JBoss)=> MessageDriven Beans (in JBoss)
Best regards,
Vincent Yeh -
3. Re: Ping time out
stephanenicoll Sep 23, 2003 8:29 AM (in response to vincentyeh)are you caching the jms session in some way? For performance reason it has a sense but this should be a problem after some time.
Regards,
Stephane -
4. Re: Ping time out
adrian.brock Sep 23, 2003 10:01 AM (in response to vincentyeh)What error do you get on the server?
Do you have enough threads configured for tomcat/jetty
to handle the requests?
Regards,
Adrian -
5. Re: Ping time out
vincentyeh Sep 23, 2003 7:37 PM (in response to vincentyeh)Hi, Stephane:
Thanks for your response :)
I create JMS session in servlet, and I don't cache any session.
======================================
InitialContext ctx = new InitialContext();
QueueConnectionFactory qcf = (QueueConnectionFactory) ctx.lookup("ConnectionFactory");
QueueConnection qConn = qcf.createQueueConnection();
QueueSession qSession = qConn.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
Queue queue = (Queue) ctx.lookup("queue/B");
QueueSender qSender = qSession.createSender(queue);
MapMessage mapMsg = qSession.createMapMessage();
mapMsg.setLong("SendTime", System.currentTimeMillis());
.....
Queue queA = (Queue) ctx.lookup("queue/A");
mapMsg.setJMSReplyTo(queA);
.....
qSender.send(mapMsg);
======================================
Best regards,
Vincent Yeh -
6. Re: Ping time out
vincentyeh Sep 23, 2003 7:50 PM (in response to vincentyeh)Hi, Adrian:
Thanks for your response :)
After server outputted the messages as my first post messages, the MDB can't receive any message from JBossMQ.
I guest that you are maybe right :), because there are many concurrent requests sometimes.
But I don't know where to configure the thread number in Jetty/Tomcat.
Could you tell me how to configure it?
Thanks a lot,
Vincent Yeh -
7. Re: Ping time out
adrian.brock Sep 23, 2003 7:53 PM (in response to vincentyeh)server/default/deploy/jbossweb-xxx.sar/META-INF/jboss-service.xml
xxx == tomcat41 or jetty
Regards,
Adrian -
8. Re: Ping time out
vincentyeh Sep 24, 2003 9:34 PM (in response to vincentyeh)Hi, Adrian:
Thanks for your response :)
I have updated the $JBOSS_HOME/server/default/deploy/jbossweb-jetty.sar/META-INF/jboss-service.xml, and
I seted
30
1500
But it didn't work.
Can I set the connection pool size to JBossMQ?
Best regards,
Vincent Yeh -
9. Re: Ping time out
adrian.brock Sep 25, 2003 7:45 AM (in response to vincentyeh)Then threads is not the problem.
There is no such thing as a connection pool for jms.
Back to my original question, what error do you get
on the server?
Regards,
Adrian -
10. Re: Ping time out
vincentyeh Sep 25, 2003 8:34 PM (in response to vincentyeh)Hi, Adrian:
I used JMeter 1.9.1 to test program, and I set 200 threads in 1 second with 10 loop count.
The server outputted the following messages:
======================================
2003-09-26 09:47:21,614 DEBUG [org.jboss.mq.Connection] Error closing the connection
java.lang.NullPointerException
at org.jboss.mq.il.oil.OILServerIL.destroyConnection(OILServerIL.java:586)
at org.jboss.mq.il.oil.OILServerIL.connectionClosing(OILServerIL.java:338)
at org.jboss.mq.Connection.(Connection.java:256)
at org.jboss.mq.Connection.(Connection.java:315)
at org.jboss.mq.SpyConnection.(SpyConnection.java:59)
at org.jboss.mq.SpyConnectionFactory.createQueueConnection(SpyConnectionFactory.java:116)
at tw.com.elitec.smg.SMGServlet.doGet(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:360)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1714)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:507)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1664)
at org.mortbay.http.HttpServer.service(HttpServer.java:863)
at org.jboss.jetty.Jetty.service(Jetty.java:460)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:775)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:939)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:792)
at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:201)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:455)
2003-09-26 09:47:21,631 ERROR [tw.com.elitec.smg.SMGServlet] SMGServlet.doGet() JMSException
org.jboss.mq.SpyJMSException: Cannot authenticate user; - nested throwable: (java.net.ConnectExcepti
on: Connection refused)
at org.jboss.mq.Connection.authenticate(Connection.java:883)
at org.jboss.mq.Connection.(Connection.java:238)
at org.jboss.mq.Connection.(Connection.java:315)
at org.jboss.mq.SpyConnection.(SpyConnection.java:59)
at org.jboss.mq.SpyConnectionFactory.createQueueConnection(SpyConnectionFactory.java:116)
at tw.com.elitec.smg.SMGServlet.doGet(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:360)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1714)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:507)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1664)
at org.mortbay.http.HttpServer.service(HttpServer.java:863)
at org.jboss.jetty.Jetty.service(Jetty.java:460)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:775)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:939)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:792)
at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:201)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:455)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:305)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:171)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:158)
at java.net.Socket.connect(Socket.java:426)
at java.net.Socket.connect(Socket.java:376)
at java.net.Socket.(Socket.java:291)
at java.net.Socket.(Socket.java:147)
at javax.net.DefaultSocketFactory.createSocket(DashoA6275)
at org.jboss.mq.il.oil.OILServerIL.createConnection(OILServerIL.java:563)
at org.jboss.mq.il.oil.OILServerIL.checkConnection(OILServerIL.java:507)
at org.jboss.mq.il.oil.OILServerIL.authenticate(OILServerIL.java:289)
at org.jboss.mq.Connection.authenticate(Connection.java:876)
... 21 more
======================================
I set
60
2000
In My Servlet doGet() method, the catch-finally block is following:
======================================
.....
} catch (IOException e) {
request.setAttribute("javax.servlet.jsp.jspException", e);
log.error("SMGServlet.doGet() IOException", e);
} catch (JMSException e) {
log.error("SMGServlet.doGet() JMSException", e);
} catch (Exception e) {
log.error("SMGServlet.doGet() Exception", e);
} finally {
try {
if (qSender != null) { qSender.close(); }
if (qSession != null) { qSession.close(); }
if (qConn != null) { qConn.close(); }
} catch (Exception e) {
log.error("SMGServlet.doGet() Close JMS Connection Exception", e);
}
}
======================================
Best regards,
Vincent Yeh -
11. Re: Ping time out
adrian.brock Sep 26, 2003 6:42 AM (in response to vincentyeh)Looks like the server cannot cope with 200 connections
in one second?
We don't let you set the backlog, but the default is 200
so I don't see why it should fail, must be at the OS level.
Is the servlet on the same server as JMS?
You can avoid the network altogether if you lookup
java:/ConnectionFactory.
Regards,
Adrian -
12. Re: Ping time out
vincentyeh Sep 28, 2003 8:54 PM (in response to vincentyeh)Hi, Adrian:
Thanks for your responses and efforts :)
The servlet is also in the same server, how do I check OS level? ulimit -a?
=====================
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) unlimited
nofiles(descriptors) 256
vmemory(kbytes) unlimited
=====================
Sometimes there are over 200 concurrent requests in the online system, whether I should change MDB to Session Beans or not?
Best regards,
Vincent Yeh -
13. Re: Ping time out
lhankins Sep 29, 2003 10:33 PM (in response to vincentyeh)I've seen a similar issue using JBoss 3.2.1 and JDK 1.4.2_01 on WinXP.
Its very sporadic and hard to reproduce, but is definitely NOT load based.
Basically, we start getting these at some point on the server (repeating about every one minute)
----------------------------------------------------------------
(Server Side Output)
22:42:09,292 WARN [Connection]: Connection failure:
org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.io.IOException: ping timeout.
at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
... 2 more
----------------------------------------------------------------
I think I've narrowed it down to our "ModelCache" class. ModelCache is a server side singleton which also happens to be a JMS listener. Here's another error message that gets generated when we get into this state:
----------------------------------------------------------------
(Server Side Output)
22:51:39,039 ERROR [ModelCache]: exception occurred [Cannot send a message to the JMS server; - nested throwable: (java.io.IOException: Client is not connected)]
org.jboss.mq.SpyJMSException: Cannot send a message to the JMS server; - nested throwable: (java.io.IOException: Client is not connected)
at org.jboss.mq.Connection.sendToServer(Connection.java:1251)
at org.jboss.mq.SpySession.sendMessage(SpySession.java:699)
at org.jboss.mq.SpyTopicPublisher.internalPublish(SpyTopicPublisher.java:120)
at org.jboss.mq.SpyTopicPublisher.publish(SpyTopicPublisher.java:68)
at com.interfacing.models.services.ModelCache.publishMessage(ModelCache.java:338)
at com.interfacing.models.services.ModelCache.vbosWereCheckedOut(ModelCache.java:271)
at com.interfacing.models.services.ModelRepositoryEjb.checkoutVbos(ModelRepositoryEjb.java:413)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:629)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:72)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:273)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:104)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:117)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java:322)
at org.jboss.ejb.Container.invoke(Container.java:674)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549)
at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:359)
at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261)
at sun.rmi.transport.Transport$1.run(Transport.java:148)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:144)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:225)
at org.jboss.mq.il.uil2.SocketManager.sendMessage(SocketManager.java:189)
at org.jboss.mq.il.uil2.UILServerIL.addMessage(UILServerIL.java:235)
at org.jboss.mq.Connection.sendToServer(Connection.java:1243)
... 37 more
-------------------------------------------
One of our customers has reported this bug in their production environment, but its been very hard for me to reproduce in my development environment (I did finally manage to reproduce it tonight).
One hypothesis I had was that this type of thing could occur during hot-deploy... since the ModelCache is a singleton, things get a little strange during a hot-deploy (newly deployed ear = new classloader = new copy of singleton).
Anyway - the fact that it has happened in the production environment kinda harpoons that theory though (no hot-deploys occur in production).
I did try upgrading to the UIL2 invocation layer recently in hopes it might fix this, but apparently that didn't work... -
14. Re: Ping time out
adrian.brock Sep 29, 2003 11:35 PM (in response to vincentyeh)Does your machine suffer from paging memory <-> disk?
This can break the garbage collector.
I have seen it where a gc took more than one minute
because of paging which caused the jms ping to fail
and the connection was closed.
You can check for a failure by installing an ExceptionListener
on the connection that re-establishes the connection.
If you are in the same VM as the JMS server you can use
the java:/ConnectionFactory which is just method calls and
therefore doesn't require a ping to check the connection
is still valid.
Regards,
Adrian