6 Replies Latest reply on Aug 19, 2013 9:22 PM by Paul Legato

    HornetQ delivery locks up using STOMP/2.3.0.CR2

    Matthew Gordon Newbie

      We have been running 2.3.0.CR2 and the message delivery locks up permanently from time to time. We have approximately 15 HornetQs running on the application servers that bridge to a single HornetQ that has a consumer built on the ActiveMessaging poller using STOMP 1.2. The recipient HornetQ continues to successfully receive messages (ie. MessageCount increases normally) but the ActiveMessaging consumer no longer processes any messages. We have tried restarting the consumer but it takes a kill -9 of that HornetQ before we can get any message consumption to occur again. Note that the process does not appear to do anything in response to kill without -9.

       

      I have attached a thread dump and the counts for MessageCount, DeliveringCount, and ConsumerCount as Justin suggested. We have let the MessageCount exceed 400,000 in the past before restarting HornetQ but with no success in getting messages to process again. DeliveringCount and ConsumerCount have always been 1.

       

      Sometimes we have seen the queueBusy log message: "Queue {0} was busy for more than {1} milliseconds. There are possibly consumers hanging on a network operation"

       

      This seems to make sense since we have a monitoring process that checks MessageCount. Not sure what is causing the queue to be so busy though. One Ruby consumer keeps the queue empty consistently all day long so there really isn't that much traffic even though it has bridges from a number of other queues. Is this a fairly typical setup? Could it be related to overhead maintaining the bridges? Those seem to be stable so I wouldn't think so.

       

      Thanks for any advice. Happy to provide additional information.