Singleton accessing JMS Message Queue Died, How to detect and restart

[smeverts]

We have a need to use the Singleton pattern to have only 1 instance of a bean which has a timer service/action.  On startup it configures the timer service pragmatically to fire every 15 seconds.  When the timer fires it checks an external process / service to see if that process/service has the ability to process (i.e. has the bandwidth) any requests.  If it can process the request it checks to see if there are any outstanding requests on a Message Queue.  We basically built a throttling mechanism to ensure that the external process can handle the load.  We programmatically connect to the message queue and use JMSConsumers to pull the messages off from the JMS message queue.  This will work fine for a period of time (10 minutes or so).  Under load (aprox 2500 messages are in the message queue) we see the following in the log file:

07:13:06,139 WARN  [server] HQ222172: Queue jms.queue.QueueName was busy for more than 1,000 milliseconds. There are possibly consumers hanging on a network operation

After that point in time the singleton appears to be dead.  The timer never fires again.  The only resolution at this point is to shutdown and restart the server. 

Does anyone have any thoughts on how to get more information from Wildfly as to the cause of this?  Is there a way for wildfly to detect a dead singleton and restart it?  Is it even possible to get a dead singleton?  Any other thoughts are welcome.

[mayerw01]

The HornetQ - Documentation - JBoss Community might contain helpful hints to help solving your problem.

Did you check if this problem persists when using activemq (Apache ActiveMQ ™ -- Index) instead?

[smeverts]

Thanks for the reference material, I have been wading through the docs in search of parameters that may have an effect on this problem with some limited success.  I have not tried a different message queue system yet, as HornetQ was supposed to be so much better/faster than the previous versions.  At this point I don't think that we want to go replacing the underlying message queue system, but will keep it in the back of my mind.

[mayerw01]

Btw apparently HornetQ is being migrated to ActiveMQ anyway (The HornetQ Team Blog: HornetQ Apache donation and Apache Artemis 1.0.0 release)

[jbertram]

A couple of things...

• Check out HornetQ's flow control features.  They may allow you to avoid a bunch of custom logic.
• The WARN message you are seeing indicates something is hanging (likely a consumer). I would recommend you get a few thread dumps during the time you see this message so you can see what's hanging (if anything).
• I don't know why your singleton dies.  That's more of an application server thing and I work pretty much exclusively on the message broker end of things.

[jbertram]

To be clear, the HornetQ code-base was donated to the Apache ActiveMQ project over a year ago now and has become Apache ActiveMQ Artemis.  This broker has already been integrated into the latest releases of Wildfly.

[smeverts]

Thanks!!! I didn't know that, must of overlooked it.