Message loss during shutdown or JMX stop of JMS Listener

[davesiracusa]

I'm looking into JMS message loss issues during shutdown.  I enqueued several thousand mesages into an ESB application.  While the messages were being processed I toggle(stop/start) lifecycle state for the jms listeners via JMX in order to simulate shutdown.  I noticed periodic warnings in the serverlog (listing 1).  The final tally indicated message loss.  I believe I found a bug in ESB. Basically when a doStop (JMXConsole) is called it sets state to STOPPING and immediately terminates the executor thread (MessageAwareListener.java). If the thread was currently processing a mesage bad things happen. I use transcated queues and it appears the JMS message transaction evaporates and the message is lost. I added some code to AbstractThreadedManagedLifecycle.java to remedy the issue.

AbstractThreadedManagedLifecycle.java
---------------------------------------
protected void doStop()
throws ManagedLifecycleException
{
runningLock.lock() ;
try
{
if (isRunning())
{
setRunning(ManagedLifecycleThreadStatae.STOPPING) ;
}
// Dave Siracusa -start
if (!waitUntilStopped())
{
throw new ManagedLifecycleException("Thread still active!") ;
} // Dave Siracusa - end

}
finally
{
runningLock.unlock() ;
}
}

MessageAwareListener.java
---------------------------
protected void doStop()
throws ManagedLifecycleException
{
super.doStop();
_execService.shutdown() ;
}

Listing 1
----------
2010-09-01 15:04:40,924 WARN [org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200) Unexpected error from doRun()
java.util.concurrent.RejectedExecutionException
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
at org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
at org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
at java.lang.Thread.run(Thread.java:619)

[davesiracusa]

I created the following JIRA: JBoss ESB JBESB-3473

[kconner]

It looks like there is a race for the final message, with what should be a simple fix.

I'll try to create a testcase for it tomorrow and verify this.

Kev