3 Replies Latest reply on May 14, 2014 8:11 PM by Jesper Karlsen

    Jboss 7.2 with hornetq xa transacted sometimes fail during failover

    Jesper Karlsen Newbie

      Hi all,

       

      We are currently testing failover for our application in Jboss 7.2 with HornetQ cluster active/passive in XA transacted mode. It seems like failover works at random, but often we get the following error on the client side and the one below on the server side. It seems to be stable when running transacted mode without XA. We are using Camel/Spring/activemq-jms-pool 5.9.0/XaPooledConnectionFactory. Has anyone successfully run this configuration with failover? Any help would be greatly appreciated!

       

      Cheers,

       

      Jesper

       

      13:50:48,113 ERROR [org.hornetq.core.client] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue]) HQ214007: XA start operation failed Cannot start, session is already doing work in
      a transaction XidImpl (574641522 bq:0.0.0.0.0.0.0.0.0.0.-1.-1.10.20.103.76.-72.-12.91.64.83.113.-113.22.0.0.126.113.0.0.0.0.0.0.0.0 formatID:131077 gtxid:0.0.0.0.0.0.0.0.0.0.-1.-1.10.20.103.76.-72.-12.91.64.83.11
      3.-113.22.0.0.126.111.49 base64:AAAAAAAAAAAAAP__ChRnTLj0W0BTcY8WAAB-cQAAAAAAAAAAAAAAAAAAAAAAAP__ChRnTLj0W0BTcY8WAAB-bzEHAgIA code:-6
      13:50:48,113 WARN  [com.arjuna.ats.jta] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue]) ARJUNA016061: TransactionImple.enlistResource - XAResource.start returned: XAException.X
      AER_PROTO for < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a14674c:-470ba4c0:53718f16:4316e, node_name=1, branch_uid=0:ffff0a14674c:-470ba4c0:53718f16:43170, subordinatenodename=null, eis_na
      me=unknown eis name >: javax.transaction.xa.XAException
              at org.hornetq.core.client.impl.ClientSessionImpl.start(ClientSessionImpl.java:1775) [hornetq-core-client.jar:]
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:636)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:397)
              at org.apache.activemq.jms.pool.XaConnectionPool.createSession(XaConnectionPool.java:70) [activemq-jms-pool.jar:5.9.0]
              at org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167) [activemq-jms-pool.jar:5.9.0]
              at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196) [spring-jms.jar:3.2.2.RELEASE]
              at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.access$200(AbstractPollingMessageListenerContainer.java:77) [spring-jms.jar:3.2.2.RELEASE]
              at org.springframework.jms.listener.AbstractPollingMessageListenerContainer$MessageListenerContainerResourceFactory.createSession(AbstractPollingMessageListenerContainer.java:531) [spring-jms.jar:3.2.2.RE
      LEASE]
              at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:300) [spring-jms.jar:3.2.2.RELEASE]
              at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:288) [spring-jms.jar:3.2.2.RELEASE]
              at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243) [spring-jms.jar:3.2.2.RELEASE]
              at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1096) [spring-jms.jar:3.2.2.RELEASE]
              at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1088) [spring-jms.jar:3.2.2.RELEASE]
              at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:985) [spring-jms.jar:3.2.2.RELEASE]
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
              at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
      
      
      13:50:48,114 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue]) javax.transaction.xa.XAException
      13:50:48,114 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.hornetq.core.client.impl.ClientSessionImpl.start(ClientSessionImpl.java:1775)
      13:50:48,114 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(Transa
      ctionImple.java:636)
      13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(Transa
      ctionImple.java:397)
      13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.apache.activemq.jms.pool.XaConnectionPool.createSession(XaConnectionPool.java:70)
      13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167)
      13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
      13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.access$200(Abstr
      actPollingMessageListenerContainer.java:77)
      13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.AbstractPollingMessageListenerContainer$MessageListenerC
      ontainerResourceFactory.createSession(AbstractPollingMessageListenerContainer.java:531)
      13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(Conne
      ctionFactoryUtils.java:300)
      13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExec
      ute(AbstractPollingMessageListenerContainer.java:288)
      13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecut
      e(AbstractPollingMessageListenerContainer.java:243)
      13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvo
      ker.invokeListener(DefaultMessageListenerContainer.java:1096)
      13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvo
      ker.executeOngoingLoop(DefaultMessageListenerContainer.java:1088)
      13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvo
      ker.run(DefaultMessageListenerContainer.java:985)
      13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at java.lang.Thread.run(Thread.java:662)
      13:50:50,165 WARN  [org.apache.camel.component.jms.DefaultJmsMessageListenerContainer] (Camel (camelContext) thread #20 - JmsConsumer[asynchronousPaymentProcessingQueue]) Setup of JMS message listener invoker failed for destination 'asynchronousPaymentProcessingQueue' - trying to recover. Cause: JTA transaction unexpectedly rolled back (maybe due to a timeout); nested exception is javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
      

       

       

       

      13:21:38,295 ERROR [org.hornetq.core.server] (Old I/O server worker (parentId: -1887525330, [id: 0x8f7ea62e, /10.20.103.88:5595])) HQ224029: Caught XA exception: org.hornetq.core.exception.HornetQXAException: Invalid transaction state
              at org.hornetq.core.server.impl.ServerSessionImpl.individualAcknowledge(ServerSessionImpl.java:634) [hornetq-server-2.3.0.CR1.jar:]
              at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:415) [hornetq-server-2.3.0.CR1.jar:]
              at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:616) [hornetq-core-client-2.3.0.CR1.jar:]
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client-2.3.0.CR1.jar:]
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client-2.3.0.CR1.jar:]
              at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:562) [hornetq-server-2.3.0.CR1.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client-2.3.0.CR1.jar:]
              at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty-3.6.2.Final.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client-2.3.0.CR1.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client-2.3.0.CR1.jar:]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty-3.6.2.Final.jar:]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) [rt.jar:1.6.0_30]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.6.0_30]
              at java.lang.Thread.run(Thread.java:701) [rt.jar:1.6.0_30]
      
      
      13:21:38,354 ERROR [org.hornetq.core.server] (Old I/O server worker (parentId: -1887525330, [id: 0x8f7ea62e, /10.20.103.88:5595])) HQ224030: Caught exception: HornetQException[errorType=ILLEGAL_STATE message=HQ119057: Could not find reference on consumer ID=67, messageId = 51,567,696,761 queue = jms.queue.asynchronousPaymentProcessingQueue]
              at org.hornetq.core.server.impl.ServerConsumerImpl.acknowledge(ServerConsumerImpl.java:645) [hornetq-server-2.3.0.CR1.jar:]
              at org.hornetq.core.server.impl.ServerSessionImpl.acknowledge(ServerSessionImpl.java:624) [hornetq-server-2.3.0.CR1.jar:]
              at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:269) [hornetq-server-2.3.0.CR1.jar:]
              at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:616) [hornetq-core-client-2.3.0.CR1.jar:]
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client-2.3.0.CR1.jar:]
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client-2.3.0.CR1.jar:]
              at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:562) [hornetq-server-2.3.0.CR1.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client-2.3.0.CR1.jar:]
              at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty-3.6.2.Final.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client-2.3.0.CR1.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client-2.3.0.CR1.jar:]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.2.Final.jar:]
              at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty-3.6.2.Final.jar:]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) [rt.jar:1.6.0_30]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.6.0_30]
              at java.lang.Thread.run(Thread.java:701) [rt.jar:1.6.0_30]
      
        • 1. Re: Jboss 7.2 with hornetq xa transacted sometimes fail during failover
          Jesper Karlsen Newbie

          Hi all,

           

          We are currently testing failover for our application in Jboss 7.2 with HornetQ cluster active/passive in XA transacted mode. It seems like failover works at random, but often we get the following error on the client side and the one below on the server side. It seems to be stable when running transacted mode without XA. We are using Camel/Spring/activemq-jms-pool 5.9.0/XaPooledConnectionFactory. Has anyone successfully run this configuration with failover? Any help would be greatly appreciated!

           

           

          My interpretation of what happens is this.

           

           

          1. Active and Passive queue servers are running

          2. We shut down the active queue server.

          3. Our application tries to continue and is given a session by activemq pool.

          4. The queue server recognizes the session as already being in use by another transaction and throws an exception.

           

           

          However - the session/transaction never seem to time out on the queue server and is also never discarded on the activemq session pool?

           

           

          Please...any help will be greatly appreciated!

           

          Cheers,

           

          Jesper

          • 2. Re: Jboss 7.2 with hornetq xa transacted sometimes fail during failover
            Andy Taylor Master

            I am a bit confused, if you are using active-mq then this is the wrong forum, however your stack trace is HornetQ.

             

            usually the reason you see that error is if you are using a session from multiple threads

            • 3. Re: Jboss 7.2 with hornetq xa transacted sometimes fail during failover
              Jesper Karlsen Newbie

              Hi Andy,

               

              We are only using the activemq PooledConnectionFactory to connect to Hornetq.

               

              As i see it there are two errors in this scenario. I would expect Hornetq Queue server to eventually timeout/discard the session in question, and also I would expect the PooledConnectionFactory to discard the session in question from its cache if an exception is thrown. Neither is happening it seems, but of course in this forum only the Hornetq part is relevant.

               

              Cheers,

              Jesper