1 Reply Latest reply on Dec 5, 2014 9:10 AM by akhettar

    PooledSession throw Exception at closing, Fabric never recovers until the the container is restarted

    akhettar Newbie

      Hi

       

      I am posting this issue here and it may not be directly related to Jboss Fuse, but I am sure many of the active members in this forum are familiar with Camel, Fabric and ActiveMQ. Hence hoping I can find an answer to the problem I am facing. I have a collection of bundles which all have an activeMQ endpoint. The issue manifest when I restart activeMA server (using remote but running locally, not embedded in a Fabric container), the PooledSession throw an Exception at closing session - see logs below. Certain connections gets refreshed successfully but some not. NO matter how many times I restart AMQ server, still seeing warnings and exceptions in the log. But if I restart the bundles individually or simply restart the container or the whole fabric server, everything get back to normal.

       

      I've come across couple of Jira tickets (https://issues.apache.org/jira/browse/AMQ-3988, https://issues.apache.org/jira/browse/AMQ-4366) depicting the same problem and stating that the issue has been fixed in version 5.9.0 of AMQ. There were some posts on StackOverflow (http://stackoverflow.com/questions/15994367/activemqs-messages-being-redelivered-pooledconnectionfactory-misconfig-suspe…) suggesting that I should set the idleTimeout to zero, but that didn't do the trick. See below details of my running environment:

       

       

      DETAILS OF MY RUNNING ENVIRONMENT

       

      OS: os x yosemite and Centos 6.

      Fabric: 1.1.0.CR5

      AMQ: apache-activemq-5.9.1 / apache-activemq-5.10.0

       

      ActiveMQ blueprint set up

       

      <blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0"
         xsi:schemaLocation="
        http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
        http://www.osgi.org/xmlns/blueprint-ext/v1.1.0 https://svn.apache.org/repos/asf/aries/tags/blueprint-0.3.1/blueprint-core/src/main/resources/org/apache/aries/blueprint/ext/blueprint-ext.xsd
        http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd
        http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0 http://svn.apache.org/repos/asf/aries/trunk/blueprint/blueprint-cm/src/main/resources/org/apache/aries/blueprint/compendium/cm/blueprint-cm-1.1.0.xsd">
      
      
         <!-- *********************************************************************************************************************
        ******************************************************************************************** ActiveMQ Configuration
        -->
      
        <!-- ActiveMQ Configuration -->
      
        <!-- define our activemq component -->
         <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
        <property name="connectionFactory" ref="poolConnectionFactory"/>
         <!-- define the jms consumer/producer as transacted -->
         <property name="transacted" value="true"/>
         <!-- setup the transaction manager to use -->
         <property name="transactionManager" ref="jmsTransactionManager"/>
        <property name="cacheLevelName" value="CACHE_CONSUMER" />
        </bean>
      
         <!-- setup spring jms TX manager -->
         <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
        <property name="connectionFactory" ref="poolConnectionFactory"/>
        </bean>
      
         <!-- setup JMS connection factory -->
         <bean id="poolConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop">
        <property name="maxConnections" value="${activemq.maxConnections}"/>
        <property name="connectionFactory" ref="jmsConnectionFactory"/>
        <property name="idleTimeout" value="0"/>
        </bean>
      
        <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
        <property name="brokerURL" value="${activemq.brokerURL}"/>
        </bean>
      
        <bean id="required" class="org.apache.camel.spring.spi.SpringTransactionPolicy">
        <property name="transactionManager" ref="jmsTransactionManager"/>
        <property name="propagationBehaviorName" value="PROPAGATION_REQUIRED"/>
        </bean>
      
      </blueprint>
      

       

       

      FABRIC LOGS

       

      2014-12-04 14:16:40,931 | INFO  | persist.payload] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:40,945 | WARN  | udit.redelivery] | PooledSession                    | ?                                   ? | 199 - org.apache.activemq.activemq-osgi - 5.10.0 | Caught exception trying rollback() when putting session back into the pool, will invalidate. javax.jms.IllegalStateException: The Session is closed

      javax.jms.IllegalStateException: The Session is closed

        at org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:767)[199:org.apache.activemq.activemq-osgi:5.10.0]

        at org.apache.activemq.ActiveMQSession.rollback(ActiveMQSession.java:588)[199:org.apache.activemq.activemq-osgi:5.10.0]

        at org.apache.activemq.jms.pool.PooledSession.close(PooledSession.java:116)[199:org.apache.activemq.activemq-osgi:5.10.0]

        at org.springframework.jms.connection.JmsResourceHolder.closeAll(JmsResourceHolder.java:215)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.connection.JmsTransactionManager.doCleanupAfterCompletion(JmsTransactionManager.java:257)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:1009)[233:org.apache.servicemix.bundles.spring-tx:3.2.8.RELEASE_1]

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:805)[233:org.apache.servicemix.bundles.spring-tx:3.2.8.RELEASE_1]

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724)[233:org.apache.servicemix.bundles.spring-tx:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1101)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1093)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:990)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_60]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_60]

        at java.lang.Thread.run(Thread.java:745)[:1.7.0_60]

      2014-12-04 14:16:40,948 | WARN  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.audit.redelivery' - trying to recover. Cause: Could not commit JMS transaction; nested exception is javax.jms.IllegalStateException: The Session is closed

      2014-12-04 14:16:40,949 | INFO  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:42,828 | WARN  | n.mb.core.error] | PooledSession                    | ?                                   ? | 199 - org.apache.activemq.activemq-osgi - 5.10.0 | Caught exception trying rollback() when putting session back into the pool, will invalidate. javax.jms.IllegalStateException: The Session is closed

      javax.jms.IllegalStateException: The Session is closed

        at org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:767)[199:org.apache.activemq.activemq-osgi:5.10.0]

        at org.apache.activemq.ActiveMQSession.rollback(ActiveMQSession.java:588)[199:org.apache.activemq.activemq-osgi:5.10.0]

        at org.apache.activemq.jms.pool.PooledSession.close(PooledSession.java:116)[199:org.apache.activemq.activemq-osgi:5.10.0]

        at org.springframework.jms.connection.JmsResourceHolder.closeAll(JmsResourceHolder.java:215)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.connection.JmsTransactionManager.doCleanupAfterCompletion(JmsTransactionManager.java:257)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:1009)[233:org.apache.servicemix.bundles.spring-tx:3.2.8.RELEASE_1]

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:805)[233:org.apache.servicemix.bundles.spring-tx:3.2.8.RELEASE_1]

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724)[233:org.apache.servicemix.bundles.spring-tx:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1101)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1093)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:990)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_60]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_60]

        at java.lang.Thread.run(Thread.java:745)[:1.7.0_60]

      2014-12-04 14:16:42,829 | WARN  | n.mb.core.error] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.error' - trying to recover. Cause: Could not commit JMS transaction; nested exception is javax.jms.IllegalStateException: The Session is closed

      2014-12-04 14:16:42,840 | INFO  | n.mb.core.error] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:43,853 | WARN  | n.mb.core.error] | PooledSession                    | ?                                   ? | 199 - org.apache.activemq.activemq-osgi - 5.10.0 | Caught exception trying rollback() when putting session back into the pool, will invalidate. javax.jms.IllegalStateException: The Session is closed

      javax.jms.IllegalStateException: The Session is closed

        at org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:767)[199:org.apache.activemq.activemq-osgi:5.10.0]

        at org.apache.activemq.ActiveMQSession.rollback(ActiveMQSession.java:588)[199:org.apache.activemq.activemq-osgi:5.10.0]

        at org.apache.activemq.jms.pool.PooledSession.close(PooledSession.java:116)[199:org.apache.activemq.activemq-osgi:5.10.0]

        at org.springframework.jms.connection.JmsResourceHolder.closeAll(JmsResourceHolder.java:215)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.connection.JmsTransactionManager.doCleanupAfterCompletion(JmsTransactionManager.java:257)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:1009)[233:org.apache.servicemix.bundles.spring-tx:3.2.8.RELEASE_1]

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:805)[233:org.apache.servicemix.bundles.spring-tx:3.2.8.RELEASE_1]

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724)[233:org.apache.servicemix.bundles.spring-tx:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1101)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1093)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:990)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_60]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_60]

        at java.lang.Thread.run(Thread.java:745)[:1.7.0_60]

      2014-12-04 14:16:43,964 | WARN  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.audit.redelivery' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:43,965 | INFO  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:44,952 | WARN  | ersist.exchange] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.persist.exchange' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:44,952 | INFO  | ersist.exchange] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:45,850 | WARN  | .mb.core.router] | PooledSession                    | ?                                   ? | 199 - org.apache.activemq.activemq-osgi - 5.10.0 | Caught exception trying rollback() when putting session back into the pool, will invalidate. javax.jms.IllegalStateException: The Session is closed

      javax.jms.IllegalStateException: The Session is closed

        at org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:767)[199:org.apache.activemq.activemq-osgi:5.10.0]

        at org.apache.activemq.ActiveMQSession.rollback(ActiveMQSession.java:588)[199:org.apache.activemq.activemq-osgi:5.10.0]

        at org.apache.activemq.jms.pool.PooledSession.close(PooledSession.java:116)[199:org.apache.activemq.activemq-osgi:5.10.0]

        at org.springframework.jms.connection.JmsResourceHolder.closeAll(JmsResourceHolder.java:215)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.connection.JmsTransactionManager.doCleanupAfterCompletion(JmsTransactionManager.java:257)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:1009)[233:org.apache.servicemix.bundles.spring-tx:3.2.8.RELEASE_1]

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:805)[233:org.apache.servicemix.bundles.spring-tx:3.2.8.RELEASE_1]

        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724)[233:org.apache.servicemix.bundles.spring-tx:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1101)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1093)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:990)[232:org.apache.servicemix.bundles.spring-jms:3.2.8.RELEASE_1]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_60]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_60]

        at java.lang.Thread.run(Thread.java:745)[:1.7.0_60]

      2014-12-04 14:16:45,851 | WARN  | .mb.core.router] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.router' - trying to recover. Cause: Could not commit JMS transaction; nested exception is javax.jms.IllegalStateException: The Session is closed

      2014-12-04 14:16:45,860 | INFO  | .mb.core.router] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:45,943 | WARN  | persist.payload] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.persist.payload' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:45,945 | INFO  | persist.payload] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:46,975 | WARN  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.audit.redelivery' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:46,975 | INFO  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:48,859 | WARN  | n.mb.core.error] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.error' - trying to recover. Cause: Could not commit JMS transaction; nested exception is javax.jms.IllegalStateException: The Session is closed

      2014-12-04 14:16:48,870 | INFO  | n.mb.core.error] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:48,962 | WARN  | ersist.exchange] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.persist.exchange' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:48,962 | INFO  | ersist.exchange] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:49,994 | WARN  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.audit.redelivery' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:49,994 | INFO  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:50,955 | WARN  | persist.payload] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.persist.payload' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:50,956 | ERROR | persist.payload] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Could not refresh JMS Connection for destination 'urn.mb.core.persist.payload' - retrying in 5000 ms. Cause: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:53,010 | WARN  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.audit.redelivery' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:53,010 | INFO  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:53,983 | WARN  | ersist.exchange] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.persist.exchange' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:53,983 | INFO  | ersist.exchange] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:54,924 | WARN  | .mb.core.router] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.router' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:54,924 | INFO  | .mb.core.router] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:55,961 | ERROR | persist.payload] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Could not refresh JMS Connection for destination 'urn.mb.core.persist.payload' - retrying in 5000 ms. Cause: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:56,029 | WARN  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.audit.redelivery' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:56,029 | INFO  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:59,000 | WARN  | ersist.exchange] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.persist.exchange' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:59,000 | INFO  | ersist.exchange] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:16:59,045 | WARN  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.audit.redelivery' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:16:59,045 | INFO  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:17:00,948 | WARN  | .mb.core.router] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.router' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:17:00,949 | INFO  | .mb.core.router] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:17:00,963 | ERROR | persist.payload] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Could not refresh JMS Connection for destination 'urn.mb.core.persist.payload' - retrying in 5000 ms. Cause: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:17:02,052 | WARN  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.audit.redelivery' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:17:02,053 | INFO  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:17:04,024 | WARN  | ersist.exchange] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.persist.exchange' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:17:04,024 | INFO  | ersist.exchange] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:17:05,064 | WARN  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.audit.redelivery' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:17:05,064 | INFO  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:17:05,966 | ERROR | persist.payload] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Could not refresh JMS Connection for destination 'urn.mb.core.persist.payload' - retrying in 5000 ms. Cause: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:17:06,976 | WARN  | .mb.core.router] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.router' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:17:06,976 | INFO  | .mb.core.router] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:17:08,078 | WARN  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.audit.redelivery' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:17:08,079 | INFO  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:17:09,045 | WARN  | ersist.exchange] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.persist.exchange' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:17:09,045 | INFO  | ersist.exchange] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:17:10,968 | ERROR | persist.payload] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Could not refresh JMS Connection for destination 'urn.mb.core.persist.payload' - retrying in 5000 ms. Cause: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:17:11,085 | WARN  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.audit.redelivery' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:17:11,085 | INFO  | udit.redelivery] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:17:12,999 | WARN  | .mb.core.router] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Setup of JMS message listener invoker failed for destination 'urn.mb.core.router' - trying to recover. Cause: Could not create JMS transaction; nested exception is javax.jms.IllegalStateException: org.apache.activemq.ConnectionFailedException: The JMS connection has failed: java.io.EOFException

      2014-12-04 14:17:12,999 | INFO  | .mb.core.router] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Successfully refreshed JMS Connection

      2014-12-04 14:17:14,072 | WARN  | ersist.exchange] | faultJmsMessageListenerContainer | ?                                   ? | 232 - org.apache.servicemix.bundles.spring-jms - 3.2.8.RELEASE_1 | Se