Topic stacks messages and cannot recover when subscribers disconnect/reconnect (2.1.2)

[steven.landers]

Environment Summary

HornetQ version = 2.1.2

Container = JBoss 5.1.0

This topic receives approximately 1 million messages per hour, and distributes to 5-8 subscribers.

Issue Summary:

1) Durable Topic is functioning correctly, and is distributing messages to non-durable subscribers

2) Non-Durable subscribers are restarted (their jboss is restarted), messages start to stack on topic (observed via MessageCount)

3) Stacking messages on topic start to page to disk after max-size-bytes is reached

4) Non-Durable subscribers are returned to service and resubscribe

5) Messages continue to pile up on the topic forever, and no messages are received by any consumer

Interestingly, we've seen the topic recover correctly before, but in those cases, the buildup didn't page to disk because volume was lower at those times.  Could paging be causing this issue?  We weren't sure if the paging was just a symptom of message stacking or a possible cause.  Is there a known issue around Topics, reconnecting non-durable subscribers, and paging?  Any recommendations?   Let me know if there is any other information that can help.  Thank you for your time.

-------------------------------------------- Paging Config

Paging configuration:

<address-setting match="#">
      <expiry-address>jms.queue.ExpiryQueue</expiry-address>
      <redelivery-delay>0</redelivery-delay>
      <max-size-bytes>104857600</max-size-bytes>
      <page-size-bytes>10485760</page-size-bytes>
      <address-full-policy>PAGE</address-full-policy>
</address-setting>

<address-setting match="jms.#">
      <redelivery-delay>10000</redelivery-delay>
      <max-delivery-attempts>10</max-delivery-attempts>
      <redistribution-delay>0</redistribution-delay>
</address-setting>

[ataylor]

could you try with a later version, weve fixed some stuff on paging since then, also check you have no stale subscribers that could cause starvation

[steven.landers]

Thanks Andy -  We're looking to upgrade asap (new environment config is in QA).   For now, is there a good way to tell if a subscriber is "stale"?  Can you explain the starvation scenario?   I'm imagining a topic distributing to a temporary queue with no consumers, or to a queue that 'thinks it has consumers' - but I may have it wrong.  Interestingly, the pile up is on the topic, not the child queue - not sure if that is relevant.

[ataylor]

ok, firstly we dont use Topics on the server, its really just a facade for JMS, we only have addresses, queues and consumers. A topic is bsically an address, i.e. topic.jms.myTopic, that has a queue for each subscription, note that the queue is durable for durable subscriptions and temporary for non durable subscribers. Now starvation is to do with flow control and paging, lets assume you are using paging which is the default, if a durable subscriber is created but then disconnects the queue still exists, now since paging occurs before routing we will get to a point where the address is full and paging (or other flow control) kicks in. At this point no more messages are routed to the queue, this is fine for this subscriber but any other subscribers on the address will also not receive any messages, this is starvation.

You can use the admin or jmx console to see if there are any queues/subscribers left.