1 2 Previous Next 15 Replies Latest reply on Aug 31, 2010 12:50 PM by Clebert Suconic

    Timed out waiting for response when sending packet

    Levona Hershtik Newbie

      Hi,

       

      I'm running the 2.1.2 version of the hornetq.

       

      I'm getting this exception when I try to send several message to a full blocking queue with transaction.

      The session is transactional, and I get the exception when commiting the session (after sending several messages)

       

      javax.jms.JMSException: Timed out waiting for response when sending packet 43
          at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)
          at org.hornetq.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:514)
          at org.hornetq.core.client.impl.DelegatingSession.commit(DelegatingSession.java:156)
          at org.hornetq.jms.client.HornetQSession.commit(HornetQSession.java:229)

       

      I was wondering, is my configuration correct? Should I even use transactions on a blocking queue?

      Does the commit supposed to throw an exception and not block on the queue?

       

      Thanks

      Levona

        • 1. Re: Timed out waiting for response when sending packet
          Clebert Suconic Master

          A Commit is supposed to be blocked.

           

          How many threads are you doing and how many transactions?

           

           

          Maybe your server is out of resources (CPU, maybe) and you may need to configure producer flow control properly. (Credits).

          • 2. Re: Timed out waiting for response when sending packet
            Levona Hershtik Newbie

            I have 10 threads performing send on a producer while 1 thread is committing the session periodicly.

            I have about 30 sessions open on 10 different queues, and each session has only 1 producer on it.

             

            I can change the flow and work with a non-transactional session, but I was trying to understand what causes the time out.

            • 3. Re: Timed out waiting for response when sending packet
              Clebert Suconic Master

              Use less credits in your producer. You are sending more messages than your server can handle (Network, memory, CPU, disk.. etc)

               

              This will give you more information : http://hornetq.sourceforge.net/docs/hornetq-2.1.2.Final/user-manual/en/html/flow-control.html#d0e3784

              • 4. Re: Timed out waiting for response when sending packet
                Levona Hershtik Newbie

                If, in fact, this is the case, then it won't matter if I will work with a non-transactional session.

                Isn't that right?

                • 5. Re: Timed out waiting for response when sending packet
                  Leos Bitto Novice

                  You should not share the jms session and its producers between threads during the transaction. Only one thread should use the producers to send the messages and then commit the session.

                  • 6. Re: Timed out waiting for response when sending packet
                    Tim Fox Master

                    Leos Bitto wrote:

                     

                    You should not share the jms session and its producers between threads during the transaction. Only one thread should use the producers to send the messages and then commit the session.

                    +1

                     

                    Leos is spot on.

                     

                    If you allow multiple threads to use a session concurrently, all sorts of strange things might start happening.

                    • 7. Re: Timed out waiting for response when sending packet
                      Levona Hershtik Newbie

                      Ok, I'll use all of your useful remarks and update you.

                       

                      Thanks

                      • 8. Re: Timed out waiting for response when sending packet
                        Levona Hershtik Newbie

                        Hi again

                         

                        I changed the producer window size and made sure there is only one thread working on each session.

                        I still have the same exception when committing the session after consuming a message.

                         

                        javax.jms.JMSException: Timed out waiting for response when sending packet 43
                            at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)
                            at org.hornetq.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:514)
                            at org.hornetq.core.client.impl.DelegatingSession.commit(DelegatingSession.java:156)
                            at org.hornetq.jms.client.HornetQSession.commit(HornetQSession.java:229)

                         

                        (Is it a coincidence it always happens on the 43rd packet?)

                         

                        I use transaction so I can manually abort a message using the rollback on a session.

                        Is there a way to do that on a non-transactional session?

                         

                        Any suggestions?

                        Thanks

                        • 9. Re: Timed out waiting for response when sending packet
                          Clebert Suconic Master

                          > (Is it a coincidence it always happens on the 43rd packet?)

                           

                          43 is the ID of the package sent on our protocol. On that case, the commit

                           

                           

                          We do transactions the time. You must be doing something obviously wrong. Since it seems you're not even stressing your server to get this message. Look at the examples.. there are several cases where we do this.

                           

                           

                          If you think there's a bug, you should share a testcase.

                          • 10. Re: Timed out waiting for response when sending packet
                            Levona Hershtik Newbie

                            Hi,

                             

                            Still trying to understand why do I have this exception. Trying to understand what I'm doing wrong and maybe create a testcase for you.

                             

                            I do have a question though. It was said before that I should not use a producer in more then one thread. Should the same thread be the one creating the session?

                            Can I create the session in one thread and use it in another? (only one thread uses it after creation).

                             

                            Thanks again

                            Levona

                            • 11. Re: Timed out waiting for response when sending packet
                              Leos Bitto Novice

                              Levona Hershtik wrote:

                               

                              Can I create the session in one thread and use it in another? (only one thread uses it after creation).

                               

                               

                              I guess that this should be fine if you ensure that creation of the session in one thread happens-before using the session in another thread. Since you use threads, you know what happens-before means in Java, right?

                              • 12. Re: Timed out waiting for response when sending packet
                                Levona Hershtik Newbie

                                Hi again,

                                 

                                I seem to have found the solution for my problem.

                                To understand the problem I added ExceptionListener on the connection (Which I should have used before). It seems like after a while running I get this exception:

                                 

                                javax.jms.JMSException: HornetQException[errorCode=2 message=Channel disconnected]
                                    at org.hornetq.jms.client.HornetQConnection$JMSFailureListener.connectionFailed(HornetQConnection.java:603)
                                    at org.hornetq.core.client.impl.FailoverManagerImpl.callFailureListeners(FailoverManagerImpl.java:768)
                                    at org.hornetq.core.client.impl.FailoverManagerImpl.failoverOrReconnect(FailoverManagerImpl.java:730)
                                    at org.hornetq.core.client.impl.FailoverManagerImpl.handleConnectionFailure(FailoverManagerImpl.java:580)
                                    at org.hornetq.core.client.impl.FailoverManagerImpl.connectionDestroyed(FailoverManagerImpl.java:291)
                                    at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:679)
                                    at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                                    at java.lang.Thread.run(Thread.java:619)
                                Caused by: HornetQException[errorCode=2 message=Channel disconnected]

                                 

                                So I looked it up and came across this post: https://jira.jboss.org/browse/HORNETQ-433

                                 

                                I tuned the connection-ttl, client-failure-check-period and producer-window-size (as Clebert recommended)

                                It seems to be helping as long as I make sure the client and server are not overwhelmed.

                                 

                                I also added re-attachment configuration to handle the closed connection, and a graceful handling of re-attachment failure. (Using ExceptionListener)

                                 

                                Thanks again for all the responses,

                                I did learn a lot from this conversation.

                                • 13. Re: Timed out waiting for response when sending packet
                                  Clebert Suconic Master

                                  You probably have an unreliable IP. We sure give a lot of stress to the server without any issues.

                                   

                                  You should probably share how you're creating your sessions.. A testcase.

                                   

                                  that would at least give us an idea on what you're doing wrong.

                                   

                                   

                                  or .. if you're not doing anything wrong.. maybe you found a bug pattern. (allthough such simple testcase would have showed up early already)

                                  • 14. Re: Timed out waiting for response when sending packet
                                    Levona Hershtik Newbie

                                    Hi Clebert,

                                    I'll try to explain my design:

                                     

                                    I have workers, which consume messages. These workers use transactions each creates it's own session as follows:

                                    Session session = connection.createSession(true, Session.SESSION_TRANSACTED);

                                    They use rollback and commit on the session according to the work they are performing. All work on a session is done in a single thread.

                                     

                                    I also have producers which collect messages to be sent to the server (one for each queue) and sends them.

                                    Each producer also has its own session created as follows:

                                    Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

                                    Here also, all work on a session is done in a single thread (Not like I did at first)

                                     

                                    A single connection is used to create all the sessions for the process.

                                     

                                    I'm working with 2.1.2

                                    My design is quite complex but if you think this might be a bug I can try to reproduce this on a simpler testcase.

                                    1 2 Previous Next