5 Replies Latest reply on Sep 26, 2017 9:32 AM by jbertram

    Stuck message in HornetQ

    nik...
    nik... Sep 21, 2017 2:38 AM

      I have a queue which contains a large number of unacknowledged messages, which are also increasing significantly. Any help how to investigate futher or any ideas how to check which are those messages.

      Sample data from JBoss admin console

      Message Count           8,605 
      Scheduled Message Count 0 
      Consumer Count          15 
      Delivering Count        8,605 
      Messages Added          3,792,745

      Other Information JBoss vesrion : JBoss-6.1.0.Final GA Edition

      I have tried PrintData class to print details of messages, but it only gives me message ids, It would be helpful if someone can suggest how to get message data

        • 1. Re: Stuck message in HornetQ
          jbertram
          jbertram Sep 21, 2017 8:01 AM (in response to nik...)

          Since your "Consumer Count" is > 0 and your "Delivering Count" (i.e. the number of messages actively being delivered to clients) is the same as your "Message Count" I can only assume that your clients stalled.  Have you investigating what your 15 clients are doing?

           

          As far as the message data goes, keep in mind that the body of a message is just an array of bytes and those bytes could be any kind of data (e.g. binary, text, etc.).  There's no reliable way to display the body of a message in a human readable form (at least not that I know of).

            • Actions
          • 2. Re: Stuck message in HornetQ
            nik...
            nik... Sep 25, 2017 12:26 AM (in response to nik...)

            Thanks Justin for the Reply.

             

            I am pretty sure that clients are not stalled,a s I can see new message getting added and delivered as "Message Added" are growing normally.

             

            Any other suggestion how to investigate further.

             

            I used some HornetQ API like PrintData from which I can see recordType 31 and 32 exists for such message and recordtype 33 is not present. Can you help me to understand what could possible cause a message to be not acknowledged.

              • Actions
            • 3. Re: Stuck message in HornetQ
              jbertram
              jbertram Sep 25, 2017 9:03 AM (in response to nik...)

              I am pretty sure that clients are not stalled,a s I can see new message getting added and delivered as "Message Added" are growing normally.

              Assuming that one or more of your consumers are stuck and your producers are not I would expect "Messages Added" to grow and even potentially for "Delivering Count" to grow as well.

               

               

              Can you help me to understand what could possible cause a message to be not acknowledged.

              The first and most obvious reason for a message to not be acknowledge is simply because the client has not acknowledged it.  Past that I couldn't say without a reproducible test-case.

               

              It's worth noting that JBoss AS 6.1 is quite old at this point.  I believe it was released back in 2011.  There's been numerous JBoss AS/Wildfly releases as well as HornetQ/Artemis releases since then.  It could be that you're hitting a bug of some kind (although I have no reason to suspect that at this point) which has already been fixed in the last 6 years.

                • Actions
              • 4. Re: Stuck message in HornetQ
                nik...
                nik... Sep 25, 2017 10:53 PM (in response to nik...)

                Assuming that one or more of your consumers are stuck and your producers are not I would expect "Messages Added" to grow and even potentially for "Delivering Count" to grow as well.

                Thanks for explanation Justin, I am actually not an expert on this matter so can you please help me in how to investigate this further, that if one or more consumer are stalled.

                 

                It's worth noting that JBoss AS 6.1 is quite old at this point.  I believe it was released back in 2011.  There's been numerous JBoss AS/Wildfly releases as well as HornetQ/Artemis releases since then.  It could be that you're hitting a bug of some kind (although I have no reason to suspect that at this point) which has already been fixed in the last 6 years.

                At the moment I am facing this issue in a live production system, where I might not be able to upgrade JBoss version, More over if I am able to find the Root Cause or any document that it was a bug fixed in JBoss , and if there is any patch available on that version. I may be able to convince client to apply patch.

                 

                Apart from above as per your suggestion, I did some investigation on logs where I am finding below

                 

                2017-09-25 18:59:20,737 ERROR [org.hornetq.ra.inflow.HornetQMessageHandler] (Thread-5846 (group:HornetQ-client-global-threads-240230078)) Failed to deliver message

                        at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:278)

                 

                and

                 

                Caused by: java.lang.InterruptedException

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1302)

                    at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:383)

                 

                Do you think this may be the cause

                  • Actions
                • 5. Re: Stuck message in HornetQ
                  jbertram
                  jbertram Sep 26, 2017 9:32 AM (in response to nik...)

                  Thanks for explanation Justin, I am actually not an expert on this matter so can you please help me in how to investigate this further, that if one or more consumer are stalled.

                  There's no single way to determine how a consumer is behaving since all consumers are different.  Assuming the consumers log something every time they consume a message you could check their logs for the expected information or lack thereof.  Past that you might could use thread dumps to see if they're functioning normally.

                   

                   

                  At the moment I am facing this issue in a live production system, where I might not be able to upgrade JBoss version, More over if I am able to find the Root Cause or any document that it was a bug fixed in JBoss , and if there is any patch available on that version. I may be able to convince client to apply patch.

                  To my knowledge, community projects are never patched, but rather new versions are simply released.  However, if you were able to identify a bug you could certainly create a patch yourself.  All the code is, after all, open source.

                   

                   

                  2017-09-25 18:59:20,737 ERROR [org.hornetq.ra.inflow.HornetQMessageHandler] (Thread-5846 (group:HornetQ-client-global-threads-240230078)) Failed to deliver message

                  at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:278)

                   

                  and

                   

                  Caused by: java.lang.InterruptedException

                  at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1302)

                  at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:383)

                   

                  Do you think this may be the cause

                  Yes, it may be the root cause.

                    • Actions