Messages stuck in the Surviving Records unable to be processed
Hi there,
Technical info:
Hornetq: v2.2.2-GA
JRE: v1.6.17
JBoss: v4.2.3 GA
OS: Windows Server 2003 R2, 32 bit
Our setup is as follows: HornetQ v2.2.2 runs on JBoss v4.2.3 GA. Other external Java clients then read from the queues setup on JBoss.
We have noticed some strange behaviour while using HornetQ v2.2.2 - upon sending hundreds of thousands of JMS messages in approximately 8 hours, some JMS messages appear to be undelivered to the client. The messages that have been undelivered are from different times in the processing. That is, there are some messages were not delivered 20 minutes into the running of the application, while others were not delivered 5 hours after the start of the application. That is to say, they were sporadic.
Running the PrintData utility over the journals and bindings reveals that the messages we have noticed were undelivered to the client have been "added" (as per operation@AddRecord;recordID=5983) and also appear in the "Surviving Records" (as per recordID=5983;userRecordType=31;isUpdate=false;Message(messageID=5983)). The output of the Journal does not show any updates or deletions on these specific messages. The output of the Journal does show updates and deletes for successfully processed messages
Our hornetq setup requires us to have 3 specific addresses for long running JMS tasks. That is, the rate at which JMS tasks are added to these queues is quicker than the rate of JMS tasks being processed by the clients. The following configuration shows this:
<address-settings>
<address-setting match="jms.queue.ExternalFtpPush">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<page-size-bytes>10485760</page-size-bytes>
<max-size-bytes>104857600</max-size-bytes>
<address-full-policy>PAGE</address-full-policy>
</address-setting>
<address-setting match="jms.queue.transcodeConsumeQueue">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<page-size-bytes>10485760</page-size-bytes>
<max-size-bytes>104857600</max-size-bytes>
<address-full-policy>PAGE</address-full-policy>
</address-setting>
<address-setting match="jms.queue.schedulerCaptionLoaderQueue">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<page-size-bytes>10485760</page-size-bytes>
<max-size-bytes>104857600</max-size-bytes>
<address-full-policy>PAGE</address-full-policy>
</address-setting>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<max-delivery-attempts>1</max-delivery-attempts>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<page-size-bytes>10485760</page-size-bytes>
<max-size-bytes>104857600</max-size-bytes>
<address-full-policy>PAGE</address-full-policy>
</address-setting>
</address-settings>
My questions are as follows:
- Why are some messages in the "Surviving Records"
- How can these messages be correctly processed?
- Why are they getting stuck there and not being processed in the first place?
- Is there a setting (or many settings) that will force these messages to be processed?
I can provide any / all configuration required. I can also provide the journals and bindings files (3MB compressed)
Kind regards,
saifeldeen
