11 Replies Latest reply on Jun 13, 2005 3:31 AM by Cho Cho Aung

    Platform dependent problem with JBossMQ

    grenaria Newbie

      We are experiencing problems with remote connections to queues and topics in JBossMQ. The problem is easy to repeat, but it only happens under Windows, under Linux we were not able to create any problems. The problem is that we had intended to have some windows based servers.

      The problem occurs both when a connection is accepted to JMS through OILServerILService and also can happen if a JBoss instance connects to another machine through the OILServerILService. By OILServerILService I mean that we tried it with the RMI, OIL, UIL2, and HTTP invokers and the problem occurs for all of those.

      As far as we can tell, at any time after the connection is accepted, even after it has been closed for some time, if you remove the network cable from the machine or lose power to the hub that the server is connected to you will start getting exceptions.

      Although the exceptions occur slightly differently they are in general the same. Generally a handful of these occur every minute after the device loses power:

      2004-03-23 14:50:23,031 WARN [org.jboss.mq.il.oil.OILServerILService] Connection failure (1).
      java.net.SocketTimeoutException: Read timed out
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
      at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2133)
      at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2313)
      at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2380)
      at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2452)
      at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2601)
      at java.io.ObjectInputStream.readByte(ObjectInputStream.java:845)
      at org.jboss.mq.il.oil.OILServerILService$Client.run(OILServerILService.java:206)
      at java.lang.Thread.run(Thread.java:534)

      Sometimes this one occurs but not always, and sometimes not for a few minutes:

      09:06:25,078 WARN [Connection] Connection failure:
      org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
      at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
      at java.lang.Thread.run(Thread.java:534)
      Caused by: java.io.IOException: ping timeout.
      at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
      ... 2 more

      If the machine that has the network failure is the initiator of the connection, when it tries to reconnect (along with all of the other exceptions it gets) it gets this one:

      09:04:35,500 ERROR [STDERR] javax.jms.JMSException: Another thread is already in receive.
      09:04:35,500 ERROR [STDERR] at org.jboss.mq.SpyMessageConsumer.receiveNoWait(SpyMessageConsumer.java:333)
      09:04:35,500 ERROR [STDERR] at com.cielo.truck.transport.TransportHandler$TransportThread.hotPotato(TransportHandler.java:173)
      09:04:35,500 ERROR [STDERR] at com.cielo.truck.transport.TransportHandler$TransportThread.loop(TransportHandler.java:279)
      09:04:35,515 ERROR [STDERR] at com.cielo.truck.transport.TransportHandler$TransportThread.run(TransportHandler.java:259)

      One time I left it running while giving exceptions every minute, and after 10 minutes or so it started giving exceptions much more often and CPU utilization went to 100%.

      I can provide code, but you can make this happen with any Windows JBoss installation and the JBossMQ client demos.

      Alex Figliolia

        • 1. Re: Platform dependent problem with JBossMQ
          grenaria Newbie

          I forgot to mention we are using jboss-3.2.3_jetty-4.2.14 and j2sdk1.4.2_01.

          Alex Figliolia

          • 2. Re: Platform dependent problem with JBossMQ
            Richard Bielak Newbie

            Are you running some firewall software on the Windows machines?

            ...richie

            • 3. Re: Platform dependent problem with JBossMQ
              grenaria Newbie

              No, no firewall software.

              • 4. Re: Platform dependent problem with JBossMQ
                Adrian Brock Master

                The ping timeout says the client is not getting a response from the server.
                This is most likely due to a network problem, i.e. the server is being told to
                connect back to the wrong client.
                If you get the same problem with UIL2 you have a more serious network problem.

                You are also using consumers concurrently which is not allowed.

                Regards,
                Adrian

                • 5. Re: Platform dependent problem with JBossMQ
                  Tim McCune Novice

                  I'm having what looks like the same problem. JBoss 3.2.3, JDK 1.4.2_02, Fedore Core 1 (client), Windows 2000 (server). I'm not using consumers concurrently. I'm not removing the machine from the network. I just start getting these messages on the client every 60 seconds after running under somewhat heavy load:

                  2004-03-29 17:08:04,087 WARN {Connection Monitor Thread} [Connection] Connection failure:
                  org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
                  at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
                  at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
                  at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
                  at java.lang.Thread.run(Thread.java:534)
                  Caused by: java.io.IOException: ping timeout.
                  at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)
                  ... 2 more

                  • 6. Re: Platform dependent problem with JBossMQ
                    Enrique Corona Newbie

                    did you find a reson for this???

                    I'm getting exactly the same error (jboss 3.2.5 debian 3.1)

                    • 7. Re: Platform dependent problem with JBossMQ
                      Mike Hemling Newbie

                      Adrian, what do you mean by "using consumers concurrently"? You cannot have multiple queue receivers connected to the same queue?

                      • 8. Re: Platform dependent problem with JBossMQ
                        NP Newbie

                        Hi,

                        I have JDK1.4.1_05 and JBoss-4.0.0. I have an application which is JMS-centric. The JBoss server cluster hosts the EJBs and JMS Qs and Topics. I have lot of client components on seperate machines which communicate with the server using JMS messages.
                        Also all the components do not have JBoss server installation. I dont use MDBs for messeging, I use Asynch message listeners and I run them on each component machine in different threads. So I have 5 to 8 listeneres from each machine to the same Q/Topic.

                        I get following exception on the components :

                        org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
                        at org.jboss.mq.Connection.asynchFailure(Connection.java:436)
                        at org.jboss.mq.Connection$PingTask.run(Connection.java:1385)
                        at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
                        at java.lang.Thread.run(Unknown Source)
                        Caused by: java.io.IOException: ping timeout.
                        at org.jboss.mq.Connection$PingTask.run(Connection.java:1377)
                        ... 2 more

                        The components recevie the messages, but it constantly throws exceptions. Any suggestions?

                        And Adrian, what do you mean by "using consumers concurrently"? You cannot have multiple queue receivers connected to the same queue?

                        Thanks in advance

                        • 9. Re: Platform dependent problem with JBossMQ
                          Stefan Heckler Novice

                          Hi,
                          I face a similar problem. I am using JBoss 3.2.6 and make heavy use of (HA)JMS within a cluster. I have written a Scheduler Mbean using org.jboss.varia.scheduler.Schedulable. This mbean is deployed as cluster-singleton using the deploy-hasingleton directory. It works fine and after failing over to a new node it works fine again. But after the failover I get the the following warning every minute:

                          2004-10-28 18:27:10,421 WARN [org.jboss.mq.Connection] Connection failure:
                          org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
                          at org.jboss.mq.Connection.asynchFailure(Connection.java:436)
                          at org.jboss.mq.Connection$PingTask.run(Connection.java:1385)
                          at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(ClockDaemon.java:364)
                          at java.lang.Thread.run(Thread.java:534)
                          Caused by: java.io.IOException: ping timeout.
                          at org.jboss.mq.Connection$PingTask.run(Connection.java:1377)


                          I have no idea, what it means (it is only a warning) , it happens only after failover even though everything works fine again.
                          Is is a ping timeout to the dead node?
                          Thanks in advance



                          • 10. Re: Platform dependent problem with JBossMQ
                            Stefan Heckler Novice

                            additional info: I am also using W2K.

                            • 11. Re: Platform dependent problem with JBossMQ
                              Cho Cho Aung Newbie

                              Hi
                              We're also having the same problem with JBoss 3.2.4 and j2sdk1.4.1
                              Can anyone help us?

                              Here is the warning message we've received:

                              org.jboss.mq.Connection - Connection failure:
                              org.jboss.mq.SpyJMSException: Connection Failed; - nested
                              throwable: (java.io.IOException: ping timeout.)
                              at org.jboss.mq.Connection.asynchFailure(Connection.java:718)
                              at org.jboss.mq.Connection$PingTask.run(Connection.java:1311)
                              at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Clock
                              Daemon.java:364) at java.lang.Thread.run(Thread.java:534)
                              Caused by: java.io.IOException: ping timeout.
                              at org.jboss.mq.Connection$PingTask.run(Connection.java:1303)