5 Replies Latest reply on Jan 22, 2015 9:40 AM by jbertram

    Problem in jms bridge connection error

    behera.smrutiranjan

      Hi 

       

      we are getting a jms bridge error it is automatically failing . and in below are the server log

      we are using jboss-eap-6.2-1

       

       

      06:09:23,719 WARN  [org.hornetq.jms.server] (Thread-18 (HornetQ-client-global-threads-1661916063)) Notified of connection failure in xa discovery, we will retry on the next recovery: HornetQException[errorType=NOT_CONNECTED message=HQ119006: Channel disconnected]

          at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]

          at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:214) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]

          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:107) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]

          at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

       

      06:09:23,719 WARN  [org.hornetq.jms.server] (Thread-20 (HornetQ-client-global-threads-1661916063)) Notified of connection failure in xa discovery, we will retry on the next recovery: HornetQException[errorType=NOT_CONNECTED message=HQ119006: Channel disconnected]

          at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]

          at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:214) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]

          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:107) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]

          at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

       

      06:09:23,720 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

      06:09:23,721 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session 7062ad7b-9ae9-11e4-aa1a-735ae9ba9623

      06:09:23,721 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session 7062ad7b-9ae9-11e4-aa1a-735ae9ba9623

      06:09:23,726 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

      06:09:23,734 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session 7073284c-9ae9-11e4-aa1a-735ae9ba9623

      06:09:23,735 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session 7073284c-9ae9-11e4-aa1a-735ae9ba9623

      06:09:23,736 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

      06:09:23,736 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session 7074aeef-9ae9-11e4-aa1a-735ae9ba9623

      06:09:23,736 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session 7074aeef-9ae9-11e4-aa1a-735ae9ba9623

      06:09:23,737 WARN  [org.hornetq.jms.server] (Thread-86) HQ122006: Detected failure on bridge connection

      06:09:24,628 WARN  [org.hornetq.jms.server] (pool-3-thread-3) HQ122004: JMS Bridge Will retry after a pause of 15,000 ms

      06:09:39,652 WARN  [org.hornetq.jms.server] (pool-3-thread-3) HQ122010: Failed to connect JMS Bridge: javax.naming.NamingException: JBAS011843: Failed instantiate InitialContextFactory com.ibm.mq.jms.context.WMQInitialContextFactory from classloader ModuleClassLoader for Module "org.hornetq:main" from local module loader @9a082e2 (finder: local module finder @8f0c85e (roots: /opt/jboss/jboss-as/modules,/opt/jboss/jboss-as/modules/system/layers/base))

          at org.jboss.as.naming.InitialContextFactoryBuilder.createInitialContextFactory(InitialContextFactoryBuilder.java:64)

          at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:664) [rt.jar:1.6.0_45]

          at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288) [rt.jar:1.6.0_45]

          at javax.naming.InitialContext.init(InitialContext.java:223) [rt.jar:1.6.0_45]

          at javax.naming.InitialContext.<init>(InitialContext.java:197) [rt.jar:1.6.0_45]

          at org.hornetq.jms.bridge.impl.JNDIFactorySupport.createObject(JNDIFactorySupport.java:55) [custom_jms-bridge-server-1.0.0.2.jar:]

          at org.hornetq.jms.bridge.impl.JNDIDestinationFactory.createDestination(JNDIDestinationFactory.java:40) [custom_jms-bridge-server-1.0.0.2.jar:]

          at org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjects(JMSBridgeImpl.java:1085) [custom_jms-bridge-server-1.0.0.2.jar:]

          at org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjectsWithRetry(JMSBridgeImpl.java:1298) [custom_jms-bridge-server-1.0.0.2.jar:]

          at org.hornetq.jms.bridge.impl.JMSBridgeImpl.access$2000(JMSBridgeImpl.java:86) [custom_jms-bridge-server-1.0.0.2.jar:]

          at org.hornetq.jms.bridge.impl.JMSBridgeImpl$FailureHandler.run(JMSBridgeImpl.java:1773) [custom_jms-bridge-server-1.0.0.2.jar:]

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]

          at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

       

       

       

       

       

      and our jms bridge configuration is:

      <jms-bridge name="EMPWmqBridge" module="ibm.wsmq">

                      <source>

                          <connection-factory name="MMQ01PRD"/>

                          <destination name="ALN.TEST.FLIFO.Q"/>

                          <context>

                              <property key="java.naming.factory.initial" value="com.ibm.mq.jms.context.WMQInitialContextFactory"/>

                              <property key="java.naming.provider.url" value="170.41.187.56:1414MMQ01PRD"/>

                          </context>

                      </source>

                      <target>

                          <connection-factory name="java:/ConnectionFactory"/>

                          <destination name="queue/PIC_STAGING"/>

                      </target>

                      <quality-of-service>AT_MOST_ONCE</quality-of-service>

                      <failure-retry-interval>15000</failure-retry-interval>

                      <max-retries>-1</max-retries>

                      <max-batch-size>10</max-batch-size>

                      <max-batch-time>10000</max-batch-time>

                      <add-messageID-in-header>false</add-messageID-in-header>

                  </jms-bridge>

        • 1. Re: Problem in jms bridge connection error
          ataylor

          this is not a HornetQ issue so you are on the wrong forum. for what its worth it looks like the issue is to do with the classpath, i.e. accessing WMQInitialContextFactory from the bridge module.

          • 2. Re: Problem in jms bridge connection error
            behera.smrutiranjan

            hi Andy

             

            what could be the issue with this error , and is there any configuration we can do to solve this error

             

            06:09:23,719 WARN  [org.hornetq.jms.server] (Thread-18 (HornetQ-client-global-threads-1661916063)) Notified of connection failure in xa discovery, we will retry on the next recovery: HornetQException[errorType=NOT_CONNECTED message=HQ119006: Channel disconnected]

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]

                at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:214) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:107) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]

                at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

            • 3. Re: Problem in jms bridge connection error
              jbertram

              Are there any other WARN or ERROR messages before this that would help explain why the connection was destroyed?  We used to have an issue where in-vm connections would be marked as idle and destroyed, but that's been solved in later version (not sure if 2.3.12 has that fix).

              1 of 1 people found this helpful
              • 4. Re: Problem in jms bridge connection error
                behera.smrutiranjan

                hi justin

                 

                thank you for replay , below are the log for the reference , i have also attached the server.log

                 

                05:27:20,894 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0af11817:5a53afe0:54ab7f88:59, node_name=1, branch_uid=0:ffff0af11817:5a53afe0:54ab7f88:5a, subordinatenodename=null, eis_name=unknown eis name >

                05:29:30,902 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0af11817:5a53afe0:54ab7f88:59, node_name=1, branch_uid=0:ffff0af11817:5a53afe0:54ab7f88:5a, subordinatenodename=null, eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@7981d79b >

                05:29:30,903 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0af11817:5a53afe0:54ab7f88:59, node_name=1, branch_uid=0:ffff0af11817:5a53afe0:54ab7f88:5a, subordinatenodename=null, eis_name=unknown eis name >

                05:31:40,908 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0af11817:5a53afe0:54ab7f88:59, node_name=1, branch_uid=0:ffff0af11817:5a53afe0:54ab7f88:5a, subordinatenodename=null, eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@50263b4e >

                05:31:40,909 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0af11817:5a53afe0:54ab7f88:59, node_name=1, branch_uid=0:ffff0af11817:5a53afe0:54ab7f88:5a, subordinatenodename=null, eis_name=unknown eis name >

                05:33:50,915 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0af11817:5a53afe0:54ab7f88:59, node_name=1, branch_uid=0:ffff0af11817:5a53afe0:54ab7f88:5a, subordinatenodename=null, eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@45487a61 >

                05:33:50,916 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0af11817:5a53afe0:54ab7f88:59, node_name=1, branch_uid=0:ffff0af11817:5a53afe0:54ab7f88:5a, subordinatenodename=null, eis_name=unknown eis name >

                05:36:00,922 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0af11817:5a53afe0:54ab7f88:59, node_name=1, branch_uid=0:ffff0af11817:5a53afe0:54ab7f88:5a, subordinatenodename=null, eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@5f66bf2f >

                05:36:00,927 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0af11817:5a53afe0:54ab7f88:59, node_name=1, branch_uid=0:ffff0af11817:5a53afe0:54ab7f88:5a, subordinatenodename=null, eis_name=unknown eis name >

                05:37:05,721 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

                05:37:05,726 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session 985decd9-a069-11e4-9261-6b09301b868e

                05:37:05,726 WARN  [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session 985decd9-a069-11e4-9261-6b09301b868e

                05:37:05,727 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

                05:37:05,728 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

                05:37:05,728 WARN  [org.hornetq.jms.server] (Thread-833 (HornetQ-client-global-threads-446717346)) Notified of connection failure in xa discovery, we will retry on the next recovery: HornetQException[errorType=NOT_CONNECTED message=HQ119006: Channel disconnected]

                    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]

                    at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:214) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]

                    at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:107) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]

                    at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

                • 5. Re: Problem in jms bridge connection error
                  jbertram

                  Looks like my suspicions were correct.  The broker is closing the in-vm connection because it believes it is idle and then when XA recovery tries to use the connection it can't.  The work to fix this issue was done on https://issues.jboss.org/browse/HORNETQ-1314.  I don't believe the fix has been ported to the 2.3.x branch where 2.3.12.Final came from so you'll either need to apply the fix yourself or move to a later version of the application server that contains the fix (e.g. Wildfly 8.2.0.Final).