2 Replies Latest reply on Sep 9, 2010 11:04 AM by kconner

    Message loss during shutdown or JMX stop of JMS Listener

    davesiracusa
      I'm looking into JMS message loss issues during shutdown.  I enqueued several thousand mesages into an ESB application.  While the messages were being processed I toggle(stop/start) lifecycle state for the jms listeners via JMX in order to simulate shutdown.  I noticed periodic warnings in the serverlog (listing 1).  The final tally indicated message loss.  I believe I found a bug in ESB. Basically when a doStop (JMXConsole) is called it sets state to STOPPING and immediately terminates the executor thread (MessageAwareListener.java). If the thread was currently processing a mesage bad things happen. I use transcated queues and it appears the JMS message transaction evaporates and the message is lost. I added some code to AbstractThreadedManagedLifecycle.java to remedy the issue.

       

      AbstractThreadedManagedLifecycle.java
      ---------------------------------------
      protected void doStop()
      throws ManagedLifecycleException
      {
      runningLock.lock() ;
      try
      {
      if (isRunning())
      {
      setRunning(ManagedLifecycleThreadStatae.STOPPING) ;
      }
      // Dave Siracusa -start
      if (!waitUntilStopped())
      {
      throw new ManagedLifecycleException("Thread still active!") ;
      } // Dave Siracusa - end

       

      }
      finally
      {
      runningLock.unlock() ;
      }
      }

       

      MessageAwareListener.java
      ---------------------------
      protected void doStop()
      throws ManagedLifecycleException
      {
      super.doStop();
      _execService.shutdown() ;
      }

       

      Listing 1
      ----------
      2010-09-01 15:04:40,924 WARN [org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200) Unexpected error from doRun()
      java.util.concurrent.RejectedExecutionException
      at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
      at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
      at org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
      at org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
      at org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
      at java.lang.Thread.run(Thread.java:619)