14 Replies Latest reply on May 15, 2007 9:53 AM by Steven Ostrowski

    JbossMQ Error at Heavly load

    jmer Newbie

      Hi list;
      Im using jboss-3.2.2 as server. Im doing a load testing my MDB using Jmeter.My MDB does is read a Object message from a queue and query a stored procedure on sql server(select) and dump the result to a mysql database and finally create a message and send to another on which another application listened. Having configured my Thread threshold to 100 and 0 ramp-up period on Jmeter, i notice that i exncounter error during the testing. Attached is the error

      .jboss.mq.Connection] Connection failure:
      org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
      at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
      at java.lang.Thread.run(Thread.java:536)
      Caused by: java.io.IOException: ping timeout.
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
      ... 2 more
      2004-05-07 17:37:23,242 WARN [org.jboss.mq.Connection] Connection failure:
      org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable: (java.io.EOFException)
      at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
      at org.jboss.mq.il.uil2.UILClientILService.asynchFailure(UILClientILService.java:145)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:394)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:331)
      at java.lang.Thread.run(Thread.java:536)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2591)
      at java.io.ObjectInputStream.readByte(ObjectInputStream.java:837)
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:278)
      ... 1 more

      below are my *-ds.xml config


      <!-- sql to call when connection is created
      <new-connection-sql>some arbitrary sql</new-connection-sql>
      <!-- sql to call on an existing pooled connection when it is obtained from pool
      <check-valid-connection-sql>some arbitrary sql</check-valid-connection-sql>




      Many thanks;

        • 1. Re: JbossMQ Error at Heavly load
          Adrian Brock Master

          The error says the server is not responding to pings.
          Unless this is due to a network failure, it would have to be unresponsive for 60
          seconds (the default PingPeriod).

          • 2. Re: JbossMQ Error at Heavly load
            Jeff Hiatt Newbie

            I am getting the same exception. We have built a large application around message queues, and send lots of messages - one to start a document to be processed and another for each stage of processing. So for each document, there are 5 or 6 messages. After a few thousand documents, we get the exception shown below.

            Also because components are kept busy, they don't open and close their connection to the queues/topics for each document. Most of them are MDBs.

            I fail to see how this has to do with the database. Also this works just fine on Weblogic. I'm sure you aren't going to tell my your jms package is not as robust as Weblogic's, right?

            • 3. Re: JbossMQ Error at Heavly load
              Jeff Hiatt Newbie

              Just as an add - I see that several are using windows and point the finger there. We are running on several platforms. However I am getting the above error during testing on a Linux platform, running JBoss 3.2.3, and Sun's 1.4.2-b28 jvm.

              • 4. Re: JbossMQ Error at Heavly load
                jmer Newbie

                We are deleveloping our codes on windows platform and deployed them on Linux box. Just an add, a servlet receives incoming xml and send it to QueueA on which MyMDB consumes it. The sqlserver reside from other box while mysql and jboss reside on same box. I also assumed that the server (linux) is not responding beacause error appearead overnight on the logs,but when i try to send another message it got process. By the way i check on the destination queue from jmx-console and i saw 3 message stuck, maybe this is causing the error on the log. Will extending the ping period resolved this issue?

                • 5. Re: JbossMQ Error at Heavly load
                  Adrian Brock Master

                  Guys, your feedback though appreciated is useless.
                  If you want help resolving this problem, I need specifics.
                  The ping is just a simple network packet that the client sends to the server saying
                  "are you still there?"

                  The alogrithm is:

                  while (notclosed)
                   if (!receivedpong())

                  Any failure is due to network or server load problems. The fact that weblogic
                  does not try to reguarly detect server availablity is of no interest to me.

                  • 6. Re: JbossMQ Error at Heavly load
                    jmer Newbie

                    Thank you the reply , greatly appricaited.

                    • 7. Re: JbossMQ Error at Heavly load
                      gorano Newbie

                      We had a lot of problems with the "ping" or more correct "no ping" during
                      heavy load. When the processor is near 100% it seems like the ping task has lower priority than every thing else.

                      For us the following solution gave us amazing performance in our last
                      project, with a lot of JMS queues:

                      We got rid of the ping by using the JVMIL configuration "java:/ConnectionFactory" for clients in the same JVM as the JMS Server.

                      I know this won't work for all scenarios, but there is no ping and it's fast!

                      We are pushing messages on to the queue remotely using a session facade (this also seems to be a lot faster compared to RMI JMS).


                      • 8. Re: JbossMQ Error at Heavly load
                        gorano Newbie

                        In Adrians description of ping "algorithm" I guess the wait method is the problem, a separate thread waiting for 60s?.

                        Maybe the ping should move out in the JMS main thread so the ping would be forced to happen during heavy load!?


                        • 9. Re: JbossMQ Error at Heavly load
                          Adrian Brock Master

                          The algorithm is pseudo code. It is implemented a lot more efficiently in the real code.

                          Of course java:/ConnectionFactory is faster, there is no serialization.

                          The ping is minimal (a few bytes once every minute) and couldn't possibly cause
                          a bottleneck. If your server can't handle two small networks requests (ping/pong)
                          within two minutes there is something wrong.

                          • 10. Re: JbossMQ Error at Heavly load
                            gorano Newbie

                            Yes I know what was wrong...

                            Our scenario was 200,000 Hashtables created more or less at the same time, and of course we wanted those to be pushed on to the queue as
                            fast as possible.

                            In this particular case you will always get very high load on the server, no matter what server you have. A little "sleep" between each message pushed onto the queue helped the situation, and the ping/pong worked. But I'm not a friend of using Thread.sleep for such things.


                            • 11. Re: JbossMQ Error at Heavly load
                              coco Newbie


                              We have the same pb as you with Jboss 3.2.2 :
                              org.jboss.mq.SpyJMSException:Connection Failed

                              Please do you know if this pb is solve in a newer Jboss release.



                              • 14. Re: JbossMQ Error at Heavly load
                                Steven Ostrowski Newbie

                                NOTE: You can also disable the ping by setting it to 0 on the InvocationLayer MBean configuration. Or, set it higher if you will.