Wildfly 10.0.0-Final - ActiveMQ messages stuck in 'Delivering'
coenenpe Feb 3, 2017 6:28 AMHi,
I have the following configuration:
- Wildfly 10.0.0 - Final
- Full-HA replication
On our production system I notice that some messages which are available on the queue, are never consumed by our MDB. It only happens after a couple of days and upto now I'm unable to reproduce it in a simple isolated test.
When restartnig the server it runs fine for a couple of days before getting into the same state.
Other messages which are send to the queue, with the same priority and at a later timestamp, are consumed immediately.
When requesting "list-delivering-messages-as-json" on the queue via CLI, I can see all my 'lost' messages (see log below)
In the log I see that 6 consumers are listed having some messages.
"durable":true,"address":"jms.queue.jms.queue.myQueue","__AMQ_CID":"2a35fea0-e76a-11e6-bb94-cdf9f003cebb","messageID":316711,"expiration":0,"type":2,"priority":4,"userID":"ID:49a381ac-e79e-11e6-bb94-cdf9f003cebb","timestamp":1485857770768}, {"durable":true,"address":"jms.queue.jms.queue.myQueue","__AMQ_CID":"adabb8f2-e79e-11e6-bb94-cdf9f003cebb","messageID":316847,"expiration":0,"type":2,"priority":4,"userID":"ID:af21de39-e79e-11e6-bb94-cdf9f003cebb","timestamp":148585794104 <...> <...> ,"consumerName":"ServerConsumer [id=e0826d31-e707-11e6-bb94-cdf9f003cebb:e0826d32-e707-11e6-bb94-cdf9f003cebb:0, filter=null, binding=LocalQueueBinding [address=jms.queue.jms.queue.myQueue, queue=QueueImpl[name=jms.queue.jms.queueddd.myQueue, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=f2b03d8c-e4a2-11e6-bb94-cdf9f003cebb]]@649168c2]]"}, <...> <Some more messages with other consumer>
In the queue Metrics I see:
- Consumer count: 15
- Message count: 1753 (According to my own logs these have never been processed by our MDB yet, all of these messages are listed in the 'delivering' list above )
- Messaged added: 130000
- Scheduled Count: 0
My MDB config:
@MessageDriven(mappedName = "ourNotificationDispatcher", activationConfig = { @ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge"), @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"), @ActivationConfigProperty(propertyName = "destination", propertyValue = "jms/queue/myQueue") }) public class EventDispatcher implements MessageListener { public void onMessage(Message message) //Processing of message } }
Do you have any idea what could be causing this issue?
Could it be related to this problem: https://developer.jboss.org/message/953165#953165 ? Do you know this jbertram ?
Many thanks in advance!