5 Replies Latest reply on Aug 27, 2013 2:30 PM by Clebert Suconic

    Jboss 6.1.0, HornetQ 2.3.0 Session creation hangs  after fail-over

    Carolina Contiu Newbie

        I have a pair of live-backup stand-alone 2.3.0 HornetQ servers (set up on different machines, NFS as share storage).

        I use Jboss 6.1.0 and removed the embedded HornetQ server.

        I've manually upgraded hornetq libraries to 2.3.0 (jboss 6.1.0 ships with an older version).

       

        Once the fail-over occurs the thread that should connect and push data to the queues hangs at session creation.

        I waited for 40 mins and stopped (killed) jboss.

       

      This is where the thread hangs:

       

      - waiting on <0x3eab7ac1> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:318) - locked <0x67e035d8> (a java.lang.Object) org.hornetq.core.client.impl.ClientSessionImpl.setTransactionTimeout(ClientSessionImpl.java:1689) org.hornetq.ra.HornetQRAXAResource.setTransactionTimeout(HornetQRAXAResource.java:249) com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:604) com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:390) org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.enlist(TxConnectionManager.java:975) org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:729) org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:459) org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:625) org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:499) org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941) org.hornetq.ra.HornetQRASessionFactoryImpl.allocateConnection(HornetQRASessionFactoryImpl.java:832) - locked <0x267f81c2> (a java.util.HashSet) org.hornetq.ra.HornetQRASessionFactoryImpl.createSession(HornetQRASessionFactoryImpl.java:465)

       

        

       

       

        Can you please provide me any clues in how to debug this.

        Is this an appropriate way (see the below configs ) to make the client aware of the existence of the

        two servers?

        

       

        This is the connection factory:

       

      <connection-factory name="ConnectionFactory">

            <xa>true</xa>

            <connectors>

               <connector-ref connector-name="netty-connector"/>

            </connectors>

            <entries>

               <entry name="XAConnectionFactory"/>

            </entries>

             <ha>true</ha>

            <!-- Pause 1 second between connect attempts -->

            <retry-interval>1000</retry-interval>

       

            <!-- Multiply subsequent reconnect pauses by this multiplier. This can be used to

            implement an exponential back-off. For our purposes we just set to 1.0 so each reconnect

            pause is the same length -->

            <retry-interval-multiplier>1.0</retry-interval-multiplier>

       

            <!-- Try reconnecting an unlimited number of times (-1 means "unlimited") -->

            <reconnect-attempts>-1</reconnect-attempts>

            </connection-factory>

       

        I've attached the 2 servers configs.

       

        Jboss conf files:

        /deploy/jms-ds.xml

       

        <connection-factories>

          <tx-connection-factory>

            <jndi-name>jms/HornetQ/XACF</jndi-name>

            <xa-transaction/>

            <rar-name>jms-ra.rar</rar-name>

            <connection-definition>org.hornetq.ra.HornetQRAConnectionFactory</connection-definition>

            <config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Topic</config-property>

            <config-property name="ConnectorClassName" type="java.lang.String">org.hornetq.core.remoting.impl.netty.NettyConnectorFactory,org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property>

            <config-property name="ConnectionParameters" type="java.lang.String">host=192.168.8.241;port=5445,host=192.168.8.51;port=5445</config-property>

            <max-pool-size>20</max-pool-size>

              

             </tx-connection-factory>

        </connection-factories>

       

        /deploy/jms-ra.rar/META-INF

       

         <resourceadapter>

              <resourceadapter-class>org.hornetq.ra.HornetQResourceAdapter</resourceadapter-class>

              <config-property>

                 <description>

                    The transport type. Multiple connectors can be configured by using a comma separated list,

                    i.e. org.hornetq.core.remoting.impl.invm.InVMConnectorFactory,org.hornetq.core.remoting.impl.invm.InVMConnectorFactory.

                 </description>

                 <config-property-name>ConnectorClassName</config-property-name>

                 <config-property-type>java.lang.String</config-property-type>

             <!--  <config-property-value>org.hornetq.core.remoting.impl.invm.InVMConnectorFactory</config-property-value>-->

             <config-property-value>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory,org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property-value>

              </config-property>

              <config-property>

                

                 <config-property-name>ConnectionParameters</config-property-name>

                 <config-property-type>java.lang.String</config-property-type>

             <!-- <config-property-value>server-id=0</config-property-value>-->

             <config-property-value>host=192.168.8.241;port=5445,host=192.168.8.51;port=5446</config-property-value>

              </config-property>

         .............

       

        Connection Factory is injected:

       

         @Resource(mappedName = "java:/jms/HornetQ/XACF")

            private static QueueConnectionFactory connectionFactory;

       

           public static QueueConnection getQueueConnection() throws JMSException

            {

                return connectionFactory.createQueueConnection();

            }

       

        .....

       

                    queueConnection = getQueueConnection();

                    queueConnection.start();

       

                    session = queueConnection.createSession(true, Session.AUTO_ACKNOWLEDGE);

       

      I would appreciate your help/advice. Thanks.

        • 1. Re: Jboss 6.1.0, HornetQ 2.3.0 Session creation hangs  after fail-over
          Andy Taylor Master

          you need to make sure that the RA connection factory is HA and has reconnectAttempts etc set to something appropriate

          • 2. Re: Jboss 6.1.0, HornetQ 2.3.0 Session creation hangs  after fail-over
            Carolina Contiu Newbie

            I've updated ra.xml, jms-xml.ds in the following way:

             

             

            ra.xml

             

               <resourceadapter>

                   <resourceadapter-class>org.hornetq.ra.HornetQResourceAdapter</resourceadapter-class>

                   <config-property>

                      <description>

                         The transport type. Multiple connectors can be configured by using a comma separated list,

                         i.e. org.hornetq.core.remoting.impl.invm.InVMConnectorFactory,org.hornetq.core.remoting.impl.invm.InVMConnectorFactory.

                      </description>

                      <config-property-name>ConnectorClassName</config-property-name>

                      <config-property-type>java.lang.String</config-property-type>

                  <!--  <config-property-value>org.hornetq.core.remoting.impl.invm.InVMConnectorFactory</config-property-value>-->

                  <config-property-value>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory,org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property-value>

                   </config-property>

                   <config-property>

                      <description>The transport configuration. These values must be in the form of key=val;key=val;,

                         if multiple connectors are used then each set must be separated by a comma i.e. host=host1;port=5445,host=host2;port=5446.

                         Each set of params maps to the connector classname specified.

                      </description>

                      <config-property-name>ConnectionParameters</config-property-name>

                      <config-property-type>java.lang.String</config-property-type>

                  <!-- <config-property-value>server-id=0</config-property-value>-->

                  <config-property-value>host=192.168.8.241;port=5445,host=192.168.8.51;port=5446</config-property-value>

                   </config-property>

                  

                   <config-property>

                     <description>Does we support HA</description>

                     <config-property-name>hA</config-property-name>

                     <config-property-type>java.lang.Boolean</config-property-type>

                     <config-property-value>true</config-property-value>

                   </config-property>

                   <config-property>

                     <description>ReconnectAttempts</description>

                     <config-property-name>ReconnectAttempts</config-property-name>

                     <config-property-type>java.lang.Integer</config-property-type>

                     <config-property-value>-1</config-property-value>

                   </config-property>

                   <config-property>

                     <description>The retry interval</description>

                     <config-property-name>RetryInterval</config-property-name>

                     <config-property-type>java.lang.Long</config-property-type>

                     <config-property-value>1000</config-property-value>

                   </config-property>

                   <config-property>

                     <description>The retry interval multiplier</description>

                     <config-property-name>RetryIntervalMultiplier</config-property-name>

                     <config-property-type>java.lang.Double</config-property-type>

                     <config-property-value>1.0</config-property-value>

                   </config-property>

             

            jms-xml.ds

            <connection-factories>

               <tx-connection-factory>

                 <jndi-name>jms/HornetQ/XACF</jndi-name>

                 <xa-transaction/>

                 <rar-name>jms-ra.rar</rar-name>

                 <connection-definition>org.hornetq.ra.HornetQRAConnectionFactory</connection-definition>

                 <config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Topic</config-property>

                 <config-property name="ConnectorClassName" type="java.lang.String">org.hornetq.core.remoting.impl.netty.NettyConnectorFactory,org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property>

                 <config-property name="ConnectionParameters" type="java.lang.String">host=192.168.8.241;port=5445,host=192.168.8.51;port=5446</config-property>

                 <config-property name="hA" type="java.lang.Boolean">true</config-property>

                 <config-property name="ReconnectAttempts" type="java.lang.Integer">-1</config-property>

                 <config-property name="RetryInterval" type="java.lang.Long">1000</config-property>

                 <config-property name="RetryIntervalMultiplier" type="java.lang.Double">1.0</config-property>

                

              

                 

                  </tx-connection-factory>

            </connection-factories>

             

            (hornetq-jms.xml was previouly attached)

             

             

             

              Please let me know if this configuration is not properly done.

             

            My MDBs do work after the failover.The problem occurs with EJBs (stateles session beans) with injected connection factory trying to

            push messages to the queues.

            In my latest tests, the session creation thread hangs around 20-22 mins, and the following exception is thrown on the server console:

             

             

             

            2013-08-20 15:34:59,551 WARN  [com.arjuna.ats.jta] ARJUNA-16061 TransactionImple.enlistResource - XAResource.start returned: XAException.XAER_RMERR for < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffffc0a808d8:126a:52135c96:528, node_name=1, branch_uid=0:ffffc0a808d8:126a:52135c96:1569, eis_name=unknown eis name >: javax.transaction.xa.XAException

                 at org.hornetq.core.client.impl.ClientSessionImpl.start(ClientSessionImpl.java:1761) [:]

                 at org.hornetq.ra.HornetQRAXAResource.start(HornetQRAXAResource.java:83) [:]

                 at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:629) [:6.1.0.Final]

                 at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:390) [:6.1.0.Final]

                 at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.enlist(TxConnectionManager.java:975) [:6.1.0.Final]

                 at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:729) [:6.1.0.Final]

                 at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:459) [:6.1.0.Final]

                 at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:625) [:6.1.0.Final]

                 at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:499) [:6.1.0.Final]

                 at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941) [:6.1.0.Final]

                 at org.hornetq.ra.HornetQRASessionFactoryImpl.allocateConnection(HornetQRASessionFactoryImpl.java:832) [:]

                 at org.hornetq.ra.HornetQRASessionFactoryImpl.createSession(HornetQRASessionFactoryImpl.java:465) [:]

             

             

             

             

            Please let me know  if you have any hints.

            • 3. Re: Jboss 6.1.0, HornetQ 2.3.0 Session creation hangs  after fail-over
              Andy Taylor Master

              the stacktrace is probably the tx timing out because failover took so long, why it took so long tho I have no idea, if you have a test of some sort i could try to recreate it

              • 4. Re: Re: Jboss 6.1.0, HornetQ 2.3.0 Session creation hangs  after fail-over
                Carolina Contiu Newbie

                I attached an archive with my test (test/jboss config files/live-bk hornetq stand-alone servers).

                With jboss 6.1.0 + hornetq libraries to 2.3.0 final - reconnection to the queues does not work for me after the fail-over  (as described above).

                The exact scenario : run attached Client class (i.e. add 10 msgs to the queue) , stop live, run Client class again.

                 

                I re-tested the scenario with jboss 6.1.0 + original hornetq libraries ( those provided in the jboss 6.1.0 final distribution), and the fail-over scenario does work.

                I am able to reconnect to the queues and push messages. However, the issues appears when  failing-back.

                My first thread attempting to connect to the queues after the fail-back hangs (the stack is below), while this exception is displayed on the server console.

                 

                Let me know if you need something else. Thanks for your help.

                 

                 

                Thread: WorkerThread#0[192.168.8.216:40636] : priority:5, demon:false, threadId:110, threadState:TIMED_WAITING

                 

                - waiting on <0x70cdbfae> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                sun.misc.Unsafe.park(Native Method)
                java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
                java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)
                org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:256)
                - locked <0x835e3ac> (a java.lang.Object)
                org.hornetq.core.client.impl.ClientSessionImpl.setTransactionTimeout(ClientSessionImpl.java:1508)
                org.hornetq.ra.HornetQRAXAResource.setTransactionTimeout(HornetQRAXAResource.java:254)
                com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:604)
                com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:390)
                org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.enlist(TxConnectionManager.java:975)
                org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:729)
                org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:459)
                org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:625)
                org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:499)
                org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
                org.hornetq.ra.HornetQRASessionFactoryImpl.allocateConnection(HornetQRASessionFactoryImpl.java:837)
                - locked <0x32cc742> (a java.util.HashSet)
                org.hornetq.ra.HornetQRASessionFactoryImpl.createSession(HornetQRASessionFactoryImpl.java:470)
                ============
                Exception in jboss console:
                19:41:04,333 WARN  [com.arjuna.ats.jta] ARJUNA-16086 TransactionImple.enlistResource setTransactionTimeout on XAResource < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffffc0a808d8:126a:521cd63c:1e, node_name=1, branch_uid=0:ffffc0a808d8:126a:521cd63c:20, eis_name=unknown eis name > threw: XAException.XAER_RMERR: javax.transaction.xa.XAException
                    at org.hornetq.core.client.impl.ClientSessionImpl.setTransactionTimeout(ClientSessionImpl.java:1515) [:6.1.0.Final]
                    at org.hornetq.ra.HornetQRAXAResource.setTransactionTimeout(HornetQRAXAResource.java:254) [:6.1.0.Final]
                    at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:604) [:6.1.0.Final]
                    at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:390) [:6.1.0.Final]
                    at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.enlist(TxConnectionManager.java:975) [:6.1.0.Final]
                    at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:729) [:6.1.0.Final]
                    at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:459) [:6.1.0.Final]
                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:625) [:6.1.0.Final]
                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:499) [:6.1.0.Final]
                    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941) [:6.1.0.Final]
                    at org.hornetq.ra.HornetQRASessionFactoryImpl.allocateConnection(HornetQRASessionFactoryImpl.java:837) [:6.1.0.Final]
                    at org.hornetq.ra.HornetQRASessionFactoryImpl.createSession(HornetQRASessionFactoryImpl.java:470) [:6.1.0.Final]
                    at org.jboss.tutorial.stateful.bean.SyncFacade.addWO(SyncFacade.java:133) [:]
                • 5. Re: Jboss 6.1.0, HornetQ 2.3.0 Session creation hangs  after fail-over
                  Clebert Suconic Master

                  We had a few fixes on HornetQ 2.3.5 which is going on the EAP. HornetQ 2.4.0.Beta1 also has these fixes.

                   

                  Basically before a XA Session wouldn't get the XID to be resent and you could have a few XA/TM issues.