Hanging de-paging. Message-count vs. delivering-count
kzakhar Sep 20, 2012 5:46 PMHi, folks!
Below the excerpt for the HornetQ (JBoss 7, HornetQ 2.2.16) configuration I have:
<subsystem xmlns="urn:jboss:domain:messaging:1.2"> <hornetq-server> <clustered>false</clustered> <persistence-enabled>true</persistence-enabled> <journal-file-size>102400</journal-file-size> <journal-min-files>2</journal-min-files> <connectors> <in-vm-connector name="in-vm" server-id="0" /> </connectors> <acceptors> <in-vm-acceptor name="in-vm" server-id="0" /> </acceptors> <address-settings> <address-setting match="jms.topic.systemBusTopic"> <dead-letter-address>jms.queue.dlqQueue</dead-letter-address> <expiry-address>jms.queue.expiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <max-size-bytes>1048576</max-size-bytes> <page-size-bytes>204857</page-size-bytes> <address-full-policy>PAGE</address-full-policy> <message-counter-history-day-limit>2 </message-counter-history-day-limit> </address-setting> </address-settings> <jms-connection-factories> <connection-factory name="InVmConnectionFactory"> <pre-acknowledge>true</pre-acknowledge> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/ConnectionFactory" /> </entries> <consumer-window-size>0</consumer-window-size> </connection-factory> <pooled-connection-factory name="hornetq-ra"> <min-pool-size>40</min-pool-size> <max-pool-size>100</max-pool-size> <transaction mode="xa" /> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/JmsXA" /> </entries> <consumer-window-size>0</consumer-window-size> </pooled-connection-factory> </jms-connection-factories> <jms-destinations></hornetq-server> </subsystem>
systemBusTopic has 80 subscriptions with the messageSelector's.
The all of the messages being sent are ObjectMessage's, so in case of a burst, there is a great need in the paging mode.
The issue I faced: upon the burst, HornetQ goes to the paging mode for systemBusTopic and not coming back:
- after the burst is over, for the next 30 minutes there is only 38 new messages were posted to the systemBusTopic
- the new messages goes to the paging store
- old messages are not consumed from the paging store
- JBoss CLI observations for systemBusTopic:
- delivering-count == 0
- message-count > 0 and increasing while the new messages receiving (consuming?)
- list-messages-for-subscription CLI operation does not show any messages ("result" => []), though the message-count > 0 for the requested subscription
Attached the document with the more details for observations.
The one of the questions I have: why do I have the inconsistency between the delivering-count and message-count?
For my understanding:
- delivering-count - number of messages that this queue is currently delivering to consumers, i.e. (# of delivered, but not yet acknowledged) + (# of "delivering", for example due a consumer is busy)
- message-count - all messages in the queue, including the scheduled and delivered
As far as scheduled messages and CLIENT_ACKNOWLEDGE is not my case, I expect those 2 values have to be the same.
In my case, delivering-count == 0 and message-count > 0, some messages are sitting in the queue and not being processed by the consumers -> preventing the HornetQ from de-paging, because HornetQ first needs to process the oldest messages, before reading from the paging store, and in the same time those messages will never reach the consumers. Deadlock detected?
Meantime, I have not seen any messaging issues in the log files, indicating some abnormal situation.
Will appreciate your feedback and guidance.
Best regards,
Konstantin
-
paging_dir_10_12_AM.output.zip 30.2 KB
-
systemBus_messaging_info.zip 1.2 MB