6 Replies Latest reply on Dec 15, 2015 1:44 PM by Scott Everts

    Singleton accessing JMS Message Queue Died, How to detect and restart

    Scott Everts Newbie

      We have a need to use the Singleton pattern to have only 1 instance of a bean which has a timer service/action.  On startup it configures the timer service pragmatically to fire every 15 seconds.  When the timer fires it checks an external process / service to see if that process/service has the ability to process (i.e. has the bandwidth) any requests.  If it can process the request it checks to see if there are any outstanding requests on a Message Queue.  We basically built a throttling mechanism to ensure that the external process can handle the load.  We programmatically connect to the message queue and use JMSConsumers to pull the messages off from the JMS message queue.  This will work fine for a period of time (10 minutes or so).  Under load (aprox 2500 messages are in the message queue) we see the following in the log file:

       

      07:13:06,139 WARN  [server] HQ222172: Queue jms.queue.QueueName was busy for more than 1,000 milliseconds. There are possibly consumers hanging on a network operation

       

      After that point in time the singleton appears to be dead.  The timer never fires again.  The only resolution at this point is to shutdown and restart the server. 

       

      Does anyone have any thoughts on how to get more information from Wildfly as to the cause of this?  Is there a way for wildfly to detect a dead singleton and restart it?  Is it even possible to get a dead singleton?  Any other thoughts are welcome.