1 2 Previous Next 15 Replies Latest reply on Jul 9, 2012 11:52 AM by goldengate001

    Messages taking lot of time to be sent (as if they were blocked)

    zenzei2k

      I have a strange problem in a Jboss 4.2.3+HornetQ 2.2.5 installation, and i don't know where to look at.

       

      I'm sending a lot of not so big messages and the operation is almost normal, with messages of around 4k bytes sended in 0.01 seconds. I'm measuring the send time as the time it takes the call to

       

      {code}producer.send(message);{code}

       

      But sometimes happens that the same messages (4k) take a long time to be sent, around 70 seconds (without throwing any kind of warn or errors). As it happens randomly in production, I don't have the steps to reproduce it in a simple manner.

       

      I must say I have Paging and Persistence disabled, with PRE_ACKNOWLEDGE mode activated.

       

      Here is the code I use to create my server connection

      {code}

      ConnectionFactory cf = (ConnectionFactory) jndiTemplate.lookup("java:/JmsXA");

      Destination topic = (Destination) jndiTemplate.lookup("topic/testTopic");

      Connection connection = cf.createConnection();

      Session session = connection.createSession(false, HornetQJMSConstants.PRE_ACKNOWLEDGE);

      MessageProducer producer = session.createProducer(topic);

      producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);

      producer.setDisableMessageID(true);

      producer.setDisableMessageTimestamp(true);

      producer.setTimeToLive(5000);

      connection.start();

      {code}

       

      The code of the send method

       

      {code}

      public void send(byte[] payload) throws Exception

        {

          try {

            BytesMessage message = session.createBytesMessage();

            message.writeBytes(payload);

            long time = System.nanoTime();

            producer.send(message);

            log.debug("Message sended in "+((System.nanoTime()-time)/Math.pow(10, 9)));

          } catch (IllegalStateException e) {

            destroy();

            initialize();

          }

        }

      {code}

       

      And here the code of the client (I have fifty of them)

       

      {code}

      ConnectionFactory cf = (ConnectionFactory) jndiTemplate.lookup("ConnectionFactory");

      Destination destination = (Destination) jndiTemplate.lookup("topic/testTopic");

      Connection connection = cf.createConnection();

      connection.setExceptionListener(this);

      Session session = connection.createSession(false, HornetQJMSConstants.PRE_ACKNOWLEDGE);

      MessageConsumer consumer = session.createConsumer(destination);

      connection.start();

      {code}

       

      This is the global address-settings configuration

       

      {code:xml}

         <address-settings>

             <!--default for catch all-->

             <address-setting match="#">

                <!--dead-letter-address>jms.queue.DLQ</dead-letter-address-->

                <!--expiry-address>jms.queue.ExpiryQueue</expiry-address-->

                <max-delivery-attempts>1</max-delivery-attempts>

                <redelivery-delay>0</redelivery-delay>

                <max-size-bytes>10485760</max-size-bytes>      

                <page-size-bytes>1048576</page-size-bytes>        

                <last-value-queue>true</last-value-queue>        

                <message-counter-history-day-limit>10</message-counter-history-day-limit>

                <address-full-policy>PAGE</address-full-policy>

             </address-setting>

          </address-settings>

       

         <persistence-enabled>false</persistence-enabled>

      {code:xml}

       

      Anybody knows why this could be happening or where to look at to try to discover the problem? I really don't know if theres is some kind of blocking or is something else. Maybe is there any timeout I can configure to kill the send of a message if takes more than X seconds?

       

      Thanks in advance!

        • 1. Re: Messages taking lot of time to be sent (as if they were blocked)
          clebert.suconic

          SO_TIMEOUT maybe? Naggling?

          • 2. Re: Messages taking lot of time to be sent (as if they were blocked)
            zenzei2k

            Clebert Suconic escribió:

             

            SO_TIMEOUT maybe? Naggling?

             

            Sorry Clebert, what are SO_TIMEOUT or Naggling?

             

            Do you know if my problem is something common (messages taken long time to be sended, but after that sending well without timeout? Do you know why this can happen or if they are simple steps to reproduce it to give you better information?

             

            Thanks in advance for your help!

            • 3. Re: Messages taking lot of time to be sent (as if they were blocked)
              clebert.suconic

              I was referring to options on so_timeout on the manual.

               

              You are using 4.2.3, which is really old, which was provided as part of a  member and it's not really "supported" by any of us.

               

               

              This is some kind of error I haven't seen before, so I suspect it's something on the integration. I suggest you try standalone, JBoss 5, (or even better JBoss 7).

               

              if you can replicate this on a standalone application we can help you, otherwise we can't help you on 4.2.3.

              • 4. Re: Messages taking lot of time to be sent (as if they were blocked)
                zenzei2k

                Clebert, I understand the limitation of my old installation and is really strange cause there are no errors nor warns, but a big delay. I'll try make a test but i doubt I could reproduce it easily.

                 

                In the meantime I'll test your suggestions and tell you how it goes. The so_timeout is the jnp.sotimeout parameter in the manual? Is this timeout used when sending messages? And what is Naggling, another parameter?

                 

                Thanks in advance!

                • 5. Re: Messages taking lot of time to be sent (as if they were blocked)
                  zenzei2k

                  Clebert, besides my last questions of the parameter, there is one extra thing I discover. Before the message that take long time to be send, there are two or three messages (same size and time to be sent), that never arrives to any of the clients. Maybe this messages are blocking the producer and because of that later messages are taking long time to be sent?

                   

                  Sorry if the information is not enough, I'm looking everything I could cause I don't know where is the problem.

                  • 6. Re: Messages taking lot of time to be sent (as if they were blocked)
                    clebert.suconic

                    Why are you using last-value-queue?

                     

                    It seems you made a mistake on this setting, as I didn't see you mentioning it anywhere. Can you try removing that value?

                     

                    Also: why don't you keep a DLQ set? you have such a short max-redeliveries.. you may lose messages in case of redeliveries and those messages are just being dropped.

                    • 7. Re: Messages taking lot of time to be sent (as if they were blocked)
                      zenzei2k

                      I think that my comment about messages lost was misundestood, my point was that before the message that take so long to be sent, there were same messages not being arrived to the client, and maybe that messages lost has something to with the next message taking so many time to be

                       

                      But just in case, I use last-vale-queue cause only the the last message sent is relevant (if there is anyone old I could discard it cause isn't needed anymore by the clients). Because of that I also didn't need the DLQ. Maybe I misunderstood that options? I can try to remove the last-value-queue setting, but this could make the producer to hang for so many seconds?

                       

                      Also there was my question about the so_timeout you mention, that parameter is the jnp.sotimeout that the manual mention? Is this timeout used when sending messages?

                       

                      Thanks in advance for your time Clebert, and sorry for so many answers.

                      • 8. Re: Messages taking lot of time to be sent (as if they were blocked)
                        clebert.suconic

                        Try disabling last-values as a test.. maybe there's a bug... (maybe with the AS4 integration?)

                        • 9. Re: Messages taking lot of time to be sent (as if they were blocked)
                          zenzei2k

                          Ok Clebert I would make that test and tell you how it goes, and I'll also try to port my application to Jboss 7.0.2 to test it as a second option.

                           

                          Thanks you very much!

                          • 10. Re: Messages taking lot of time to be sent (as if they were blocked)
                            clebert.suconic

                            BTW: you didn't set the last-value property at your queue as specified on the manual:

                             

                            27.2. Using Last-Value Property

                            • 11. Re: Messages taking lot of time to be sent (as if they were blocked)
                              zenzei2k

                              Clebert, after some days of tests, I continue having problems.

                               

                              First of all I must say you were right, the last-value property (_HQ_LVQ_NAME) at my queue was missing. But I tried adding that property, and also removing the last-value from the hornetq-configuration.xml and the problem continues.

                               

                              After reading this thread http://community.jboss.org/message/626049?tstart=0, I also tried removing the timeToLive in the producer cause my clients sometimes doesn't have it's hours syncrhonized with the server and the ttl was very low (around 5 seconds). But that doesn't help either.

                               

                              Another test I made was adding again DLQ and ExpiryQueue, and looking wheren the problem araise if anyone has messages, but both of them were empty. I didn't have any pagination files inside hornetq/data folder. I also tried with more delivery-attempts and redelivery-delay. This was my final configuration

                               

                              {code:xml}

                              <address-settings>

                                  <address-setting match="#">

                                      <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                                  <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                                      <max-delivery-attempts>3</max-delivery-attempts>

                                      <redelivery-delay>100</redelivery-delay>

                                      <max-size-bytes>10485760</max-size-bytes>

                                      <page-size-bytes>1048576</page-size-bytes>

                                  <message-counter-history-day-limit>10</message-counter-history-day-limit>

                                      <address-full-policy>PAGE</address-full-policy>

                                   </address-setting>

                              </address-settings>

                              {code}

                               

                              In every case, the problem I have is messages of about 4kb taking long time to be sent in the server (more than 1 minute, sometimes 2 minutes when in normal operation takes same milliseconds), in the call of

                               

                               

                              {code}
                              producer.send(message);
                              {code}

                               

                               

                              In general the problem is NOT messages delivered by the server and not received in client, although in a few cases happens that two or three messages before the one that get blocks aren't received in the clients.

                               

                              If it helps, here is an extract of the log with TRACE enabled for HornetQ when I message take long time to be sent (more than 60 seconds)

                               

                               

                              -----

                               

                              2011-12-16 19:42:12,421 5863718 TRACE [org.hornetq.ra.HornetQRAManagedConnection] (Timer-2:) getSession() -> session HornetQRASession->org.hornetq.core.client.impl.DelegatingSession@b846c6

                              2011-12-16 19:42:12,421 5863718 TRACE [org.hornetq.ra.HornetQRASession] (Timer-2:) getSessionInternal HornetQRASession->org.hornetq.core.client.impl.DelegatingSession@2e749c for org.hornetq.ra.HornetQRASession@69b824

                              2011-12-16 19:42:12,421 5863718 TRACE [org.hornetq.ra.HornetQRASession] (Timer-2:) createBytesMessageHornetQRASession->org.hornetq.core.client.impl.DelegatingSession@2e749c

                               

                              2011-12-16 19:42:12,421 5863718 DEBUG [com.test.jms.HornetQMessageSender] (Timer-2:) Sending message

                               

                              2011-12-16 19:42:12,421 5863718 TRACE [org.hornetq.ra.HornetQRASession] (Timer-2:) lock()

                              2011-12-16 19:42:12,421 5863718 TRACE [org.hornetq.ra.HornetQRAManagedConnection] (Timer-2:) tryLock()

                              2011-12-16 19:42:12,421 5863718 TRACE [org.hornetq.ra.HornetQRAManagedConnectionFactory] (Timer-2:) getUseTryLock()

                              2011-12-16 19:42:12,421 5863718 TRACE [org.hornetq.ra.HornetQRAMCFProperties] (Timer-2:) getUseTryLock()

                              2011-12-16 19:42:12,421 5863718 TRACE [org.hornetq.ra.HornetQRAManagedConnection] (Timer-2:) lock()

                              2011-12-16 19:42:12,421 5863718 TRACE [org.hornetq.ra.HornetQRAMessageProducer] (Timer-2:) send org.hornetq.ra.HornetQRAMessageProducer@5edf72 message=HornetQMessage[null]:PERSISTENT

                              2011-12-16 19:43:18,203 5871500 TRACE [org.hornetq.core.security.impl.SecurityStoreImpl] (Old I/O server worker (parentId: 19009489, [id: 0x01220fd1, /172.20.2.12:5445]):) checking access permissions to jms.topic.testTopic

                              2011-12-16 19:43:18,203 5871500 TRACE [org.hornetq.ra.HornetQRAMessageProducer] (Timer-2:) sent org.hornetq.ra.HornetQRAMessageProducer@5edf72 result=HornetQMessage[null]:NON-PERSISTENT

                              2011-12-16 19:43:18,203 5871500 TRACE [org.hornetq.ra.HornetQRASession] (Timer-2:) unlock()

                              2011-12-16 19:43:18,203 5871500 TRACE [org.hornetq.ra.HornetQRAManagedConnection] (Timer-2:) unlock()

                               

                              2011-12-16 19:43:18,203 5871500 DEBUG [com.test.jms.HornetQMessageSender] (Timer-2:) Message sended in 65.778664156

                               

                              -----

                               

                               

                              I think I mention before, but I configure HornetQ topic with Pre ack mode in both server and clients with,

                               

                              {code}session = connection.createSession(false, HornetQJMSConstants.PRE_ACKNOWLEDGE){code}

                               

                              And I'm not receiving messages in the client with the receive method, but with a MessageListener (onMessage method).

                               

                              {code}

                              ...

                              consumer.setMessageListener(this);

                              ...

                              public void onMessage(Message message)

                              {

                              //doSomething

                              }
                              {code}

                               

                              Do you think there is anything else I can test? In one of your messages you mention two things I don't test cause I don't know exactly what they are, Naggling and SO_TIMEOUT. Are they used when sending messages? SO_TIMEOUT is the jnp.sotimeout parameter in the manual?

                               

                              About the Jboss migration I don't mind to switch, but I'm not truly sure if that would help. I start looking in Jboss 7 (7.0.2 and 7.1.0beta) but both of them have features missing that I use in Jboss 4.2.3. Would Jboss 6.1.0 be a right choice? Is hornetQ better integrated there?

                               

                              Thanks in advance as always Clebert!

                              • 12. Re: Messages taking lot of time to be sent (as if they were blocked)
                                zenzei2k

                                Hi Clebert, did you manage to read my last message? I'm truly don't know what else I could try or change.

                                 

                                Thanks in advance and sorry for the reask.

                                • 13. Re: Messages taking lot of time to be sent (as if they were blocked)
                                  goldengate001

                                  Nico,

                                   

                                  Did you figure this out? I am having the same issue with JBOSS 7.2.0.

                                  Any inputs would be much appreciated.

                                   

                                  Thanks!    

                                  • 14. Re: Messages taking lot of time to be sent (as if they were blocked)
                                    zenzei2k

                                    goldengate001,

                                     

                                    I don't know if this would work for you, but I attach the last configuration I have, with no blocks since a few month (until now!). The only difference between my last post is the configuration of the ConsumerWindowSize and PreAcknowledge properties for the Connection Factories (hornetq-jms.xml and jms-ds.xml) and Resource Adapter (ra.xml).

                                     

                                    But sincerely https://www.google.com.ar/search?hl=es&safe=off&client=firefox-a&hs=HUG&rls=org.mozilla:es-AR:official&sa=X&ei=Zu_6T6W3LIaQ8wSi0KSJBw&ved=0CEQQvwUoAQ&q=sincerely&spell=1I really don't understand very well if that was causing the blocks cause my jms knowledge isn't the best. The last value queue property is also configured and the property set in the message sent (as Clebert suggest).

                                     

                                    Remember that his configuration is for jboss 4.2.3 with hornetq 2.2.5 Final.

                                    1 2 Previous Next