activemq infinite msg redeliveries --- hangs after a few hours
we are currently using fuse message broker 5.3.0.5 along with fuse-esb 3.4.0.5.
We have a jms component that consumes msg out of a transacted activeMQ queue.
Since we want the msg to be redelivered indefintely if our host is down, we have set the attribute redeliveryPolicy.maximumRedeliveries of the jms queue to be -1.
We inject some msg and since the host is down, we notice that the message continues to get resent. However after around 4 hours, the redelivery stops. Looking at the log,
we notice that the messageListenerContainer seems to be stuck waiting for the notification from the camel thread.
We believe that the messageListenerContainer spawns the camel thread to process the msg. It would wait for the notification from the thread in order to commit or roll-back the transaction. Since there is no notification coming back, it just gets stuck there. The message has to be manually removed from the queue. Sometimes, ActiveMQ/Servicemix would get into a bad state (ie. stops processing msgs) and requires rebooting.
the following log shows that the messageListenerContainer is waiting for the notification to come back. It never comes out of that state.
2011-03-21 22:22:49:666 GMT-0700, DEBUG, DefaultMessageListenerContainer-1682, org.apache.servicemix.jbi.messaging.DeliveryChannelImpl - Waiting for exchange ID:10.102.173.77-12edb3e9119-11:1686 (8815a9) to be answered in DeliveryChannel{servicemix-jms} from sendSync
Meanwhile the spawn camel thread processes and sends the message out to the host. The last log statement of the camel thread shows that it has finished sending the msg out.
Question: could we run out of thread or memory causing the notification to be lost ? could it be that the garbage collector is not kicked in for activeMQ ? Or is this a bug with ActiveMQ ?
We repeated the above test but this time, monitoring the thread counts and memory usage of both servicemix and ActiveMQ using jconsole. The problem did not repeat itself ie. the redeliveries continues for over 8 hours until we reconnect the host and the msg gets delivered successfully.
Wonder whether anyone out there has encountered similar problems and could provide some useful pointers.
would appreciate it.
Following is the spring configuration of the jms consumer:
<jms:consumer service="......." ......... destinationName="rmd-ss" connectionFactory="#connectionFactory1" cacheLevel="3" transacted="xa" /> <amqpool:xa-pool id="connectionFactory2" maxConnections="8" brokerURL="${com.ventyx.ss.rmd.brokerUrl.0}" transactionManager="#transactionManager" /> <bean id="transactionManager" class="org.springframework.jndi.JndiObjectFactoryBean"> <property name="jndiName" value="java:comp/env/smx/TransactionManager" /> </bean>
Thanks