Running into a deadlock because of messageListener handling

[nomind]

Please provide me solution to avoid following scenario -

The scenario for deadlock in JMS MessageListener while closing resources in onException() method of ExceptionListener:

•     When an exception occurs in a MessageListener thread, it calls the asynchFailure() method of Session class which in turns calls the asynchFailure() method of Connection class. In the Connection class, a lock is taken on ‘elLock’ object and if there is no ExceptionListenerRunnable thread running, a ExceptionListenerRunnable thread is created and started. In the run() method of ExceptionListenerRunnable, a lock is again taken on ‘elLock’ object and the onException() method of ExceptionListener class is invoked.

•     In onException() method of ExceptionListener class we try to recreate the JMS Connection. Before reconnecting we close all existing resources (Consumer, Session and Connection). In the close() API of SpyMessageConsumer class, the Message Consumer checks for any MessageListener thread and if there is any listener thread it calls the join() API of Thread class and wait for the listener thread to die.

•     In the meanwhile if one more exception occurs in MessageListener thread, it again invokes asynchFailure() method of Connection class. If the earlier ExceptionListenerRunnable thread has not released the lock on the ‘elLock’ object, the MessageListener thread waits outside the synchronized block in asynchFailure() method of Connection class.

This leads to a deadlock situation where ExceptionListener thread is waiting for the MessageListener thread to die and MessageListener thread is waiting for the ExceptionListener class to release lock on ‘elLock’ object.

Though in asynchFailure() method of Connection class, a check is there to verify and log a warning message (“Connection failure, already in the exception listener”) if the ExceptionListenerRunnable thread is already running but this check is also inside the synchronized block. So even if ExceptionListenerRunnable thread is already running, the other thread will not be able to take the lock on ‘elLock’ object and the warning message will never be displayed. This seems to be a bug in JMS code as the check to verify an already running ExceptionListenerRunnable thread should be outside the synchronized block and if the ExceptionListenerRunnable thread is already running, the method should return from there.

Following is my thread dump -

"ExceptionListener Connection@206865307[token=ConnectionToken:ID:44/ee177c7bd4386f174c32bc563cb9b67a rcvstate=STARTED]"Id=331 in WAITING on lock=java.lang.Thread@13f1442e

    at java.lang.Object.wait(Native Method)

    at java.lang.Thread.join(Thread.java:1258)

    at java.lang.Thread.join(Thread.java:1332)

    at org.jboss.mq.SpyMessageConsumer.close(SpyMessageConsumer.java:554)

    at com.tribalfusion.jms.JMSReceiver.closeResources(JMSReceiver.java:88)

    at com.tribalfusion.jms.JMSTopicReceiver.setJMSConfig(JMSTopicReceiver.java:32)

      - locked com.tribalfusion.jms.JMSTopicNonDurableSubscriber@600877d4

    at com.tribalfusion.jms.JMSReceiver.reconfigureJMS(JMSReceiver.java:140)

    at com.tribalfusion.jms.JMSReceiver.reconnect(JMSReceiver.java:128)

    at com.tribalfusion.jms.JMSReceiver$2.onException(JMSReceiver.java:77)

    at org.jboss.mq.Connection$ExceptionListenerRunnable.run(Connection.java:1320)

      - locked java.lang.Object@4bf3308d

    at java.lang.Thread.run(Thread.java:722)

MessageListenerThread - dbSaveAckTopic"Id=45 in BLOCKED on lock=java.lang.Object@4bf3308d	owned by ExceptionListener Connection@206865307[token=ConnectionToken:ID:44/ee177c7bd4386f174c32bc563cb9b67a rcvstate=STARTED] Id=331
	at org.jboss.mq.Connection.asynchFailure(Connection.java:424)
	at org.jboss.mq.SpySession.asynchFailure(SpySession.java:1063)
	at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java:731)
	at java.lang.Thread.run(Thread.java:722)

Full thread dump is attached.