JMS messages are getting trapped inside of the HQ system. That is a publisher can still publish messages, but consumers are unable to receive those messages until some specific events happen.
Conside the following scenario:
1)Create 2 durable subscriptions called S1 and S2 on a single topic destination D1.
2)Shut down the durable consumers.
3)Publish 500 messages on destination D1. The messages should be of sufficient size to ensure that some of them get paged. In my specific case, 63 messages were in memory, and the remaining 437 got paged.
Each message was about 50K, and the settings were:
4)Create a non-durable consumer 'C1' on the same destination D1.
5) Restart durable consumer S1. You may need to wait for some time to have elapsed between steps (2) and step (5) else you may get a:
javax.jms.IllegalStateException: Cannot create a subscriber on the durable subscription since it already has subscriber(s)
Observation: Durable S1 only receives those 63 messages that are in memory. Instead it should have received all 500.
6) Publish a single message to destination D1. At this time the total number of messages published is 501.
Observation: Neither durable S1 nor does non-durable C1 receive this last message.
7) Restart durable consumer S2.
Observation: Durable S2 receives the first 63 messages (those that were in memory). And after that, both S1 and S2 start receiving the remaining (501-63) messages. Also, non-durable C1 receives 438 messages (the 437 that had been paged plus the one that was published in step 6).
Two of the consumers did not work as expected:
i) Durable consumer S1 should have received all 500 messages when it was started up in step 5, and it should have received the single message that was published in step 6 . Instead it only received those messages that were in memory.
ii) Non durable consumer C1 should have only received those messages that were published after it was created. In this case it would be the single message published in step 6. Instead it receives all paged messages + the message published after it was created.
I suspect that this behavior is related to the way the paging system works. Messages get paged and are not really associated with any JMS destination until they come back into memory. But this design has some unintended consequences. For example, we have already seen that the message count as reported back by the Queue mbean is incorrect (https://jira.jboss.org/browse/HORNETQ-31). Additionally other operations such as the "RemoveMessages" operation on the Queue mbean do not work as expected. You would expect that all messages would be removed, but in reality only those messages that were in memory get removed. You have to keep repeating the operation until you get back a result of 0 messages removed.