Bug HornetQ not delivering message when a consumer unexpectedly exits

[magnus.hagstrom]

Hi

I have been testing out message grouping in Hornetq, I found a bug when a consumer/client exits withou closing session/connection. To test you can kill the consumer ( kill -9 or similar ) . I have run this test on the latest version 2.2.5.

My scenario I tested was this. Simulating an App. Server failure. Start up jconsole to monitor number of consumer and message count.

Start hornetq

Start 3 consumers.

Send 10 msg with 3 different messagegroups, each consumer now gets 10 messages each.

So far its all good.

Now close down 1 consumer with kill -9

IMPORTANT STEP

Resend 10 msg with 3 different messagegroups again BEFORE HornetQ discovers that the consumer is dead. Monitor this in jconsole. You have a minute or 2.

Now things goes wrong. You have 2 consumers running that should get a all messages you sent. But instead HornetQ delivers some (if you are lucky ) messages. It tries to deliver to the client you killed with kill -9 but fails, it will wait for a while and then put 1 message on DLQ, but the rest of the message for that group will remain in queue and no other messages will be delivered. If you start up a new consumer it will discover this new consumer and start delivering messages again.

I guess a proper behavior should be that instead of putting that message on DLQ it should remove message from message group and assign that message group to an existing consumer.

Attached files,  a simple consumer, a simple sender.

[magnus.hagstrom]

JIRA https://issues.jboss.org/browse/HORNETQ-801

[jdahlbom]

We have a similar - but not identical - problem, although I have not been able to diagnose it as well as you did:

Our symptoms:

* Our server has been running for a while, and has received a number of messages via a JMS bridge.

* Target of that JMS bridge is queue/receiverQueue, which in turn has diverts that distribute these messages

into three queues, which we an call queue/A, queue/B and queue/C for now.

* Some of the messages from queue/A have been consumed earlier, but that seems to have stopped: No new messages are getting consumed.

* Looking at the queue/A entry in JMX console, I can see that there are messages in the queue, but also

  the Consumer Count is >1.  Usually at this point MessageCount==DeliveryCount, meaning that all the messages

  have been assigned to some consumer for consumption.

* queue/B and queue/C may at this point still have just one consumer, and work correctly.

* queue/A, B & C are identical in the sense that each of them have one consumer, created via Spring using

http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/jms/listener/DefaultMessageListenerContainer.html

* If I restart the server, the consumer count stays at 1 for a while before growing again.

Your description of dead consumers hogging messages in a queue fits our symptoms.

[clebert.suconic]

I don't think you're hitting the same issue. You probably have blocking configured on paging mode and you're running out of credits / not consuming messages. The other message on this thread was about message grouping which is a different thing.