Stuck message in HornetQ

[nik...]

I have a queue which contains a large number of unacknowledged messages, which are also increasing significantly. Any help how to investigate futher or any ideas how to check which are those messages.

Sample data from JBoss admin console

Message Count           8,605 
Scheduled Message Count 0 
Consumer Count          15 
Delivering Count        8,605 
Messages Added          3,792,745 

Other Information JBoss vesrion : JBoss-6.1.0.Final GA Edition

I have tried PrintData class to print details of messages, but it only gives me message ids, It would be helpful if someone can suggest how to get message data

[jbertram]

Since your "Consumer Count" is > 0 and your "Delivering Count" (i.e. the number of messages actively being delivered to clients) is the same as your "Message Count" I can only assume that your clients stalled.  Have you investigating what your 15 clients are doing?

As far as the message data goes, keep in mind that the body of a message is just an array of bytes and those bytes could be any kind of data (e.g. binary, text, etc.).  There's no reliable way to display the body of a message in a human readable form (at least not that I know of).

[nik...]

Thanks Justin for the Reply.

I am pretty sure that clients are not stalled,a s I can see new message getting added and delivered as "Message Added" are growing normally.

Any other suggestion how to investigate further.

I used some HornetQ API like PrintData from which I can see recordType 31 and 32 exists for such message and recordtype 33 is not present. Can you help me to understand what could possible cause a message to be not acknowledged.

[jbertram]

I am pretty sure that clients are not stalled,a s I can see new message getting added and delivered as "Message Added" are growing normally.

Assuming that one or more of your consumers are stuck and your producers are not I would expect "Messages Added" to grow and even potentially for "Delivering Count" to grow as well.

Can you help me to understand what could possible cause a message to be not acknowledged.

The first and most obvious reason for a message to not be acknowledge is simply because the client has not acknowledged it.  Past that I couldn't say without a reproducible test-case.

It's worth noting that JBoss AS 6.1 is quite old at this point.  I believe it was released back in 2011.  There's been numerous JBoss AS/Wildfly releases as well as HornetQ/Artemis releases since then.  It could be that you're hitting a bug of some kind (although I have no reason to suspect that at this point) which has already been fixed in the last 6 years.

[nik...]

Assuming that one or more of your consumers are stuck and your producers are not I would expect "Messages Added" to grow and even potentially for "Delivering Count" to grow as well.

Thanks for explanation Justin, I am actually not an expert on this matter so can you please help me in how to investigate this further, that if one or more consumer are stalled.

It's worth noting that JBoss AS 6.1 is quite old at this point.  I believe it was released back in 2011.  There's been numerous JBoss AS/Wildfly releases as well as HornetQ/Artemis releases since then.  It could be that you're hitting a bug of some kind (although I have no reason to suspect that at this point) which has already been fixed in the last 6 years.

At the moment I am facing this issue in a live production system, where I might not be able to upgrade JBoss version, More over if I am able to find the Root Cause or any document that it was a bug fixed in JBoss , and if there is any patch available on that version. I may be able to convince client to apply patch.

Apart from above as per your suggestion, I did some investigation on logs where I am finding below

2017-09-25 18:59:20,737 ERROR [org.hornetq.ra.inflow.HornetQMessageHandler] (Thread-5846 (group:HornetQ-client-global-threads-240230078)) Failed to deliver message

        at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:278)

and

Caused by: java.lang.InterruptedException

    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1302)

    at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:383)

Do you think this may be the cause

[jbertram]

Thanks for explanation Justin, I am actually not an expert on this matter so can you please help me in how to investigate this further, that if one or more consumer are stalled.

There's no single way to determine how a consumer is behaving since all consumers are different.  Assuming the consumers log something every time they consume a message you could check their logs for the expected information or lack thereof.  Past that you might could use thread dumps to see if they're functioning normally.

At the moment I am facing this issue in a live production system, where I might not be able to upgrade JBoss version, More over if I am able to find the Root Cause or any document that it was a bug fixed in JBoss , and if there is any patch available on that version. I may be able to convince client to apply patch.

To my knowledge, community projects are never patched, but rather new versions are simply released.  However, if you were able to identify a bug you could certainly create a patch yourself.  All the code is, after all, open source.

2017-09-25 18:59:20,737 ERROR [org.hornetq.ra.inflow.HornetQMessageHandler] (Thread-5846 (group:HornetQ-client-global-threads-240230078)) Failed to deliver message

at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:278)

and

Caused by: java.lang.InterruptedException

at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1302)

at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:383)

Do you think this may be the cause

Yes, it may be the root cause.