Hi all,

We are currently testing failover for our application in Jboss 7.2 with HornetQ cluster active/passive in XA transacted mode. It seems like failover works at random, but often we get the following error on the client side and the one below on the server side. It seems to be stable when running transacted mode without XA. We are using Camel/Spring/activemq-jms-pool 5.9.0/XaPooledConnectionFactory. Has anyone successfully run this configuration with failover? Any help would be greatly appreciated!

Cheers,

Jesper

13:50:48,113 ERROR [org.hornetq.core.client] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue]) HQ214007: XA start operation failed Cannot start, session is already doing work in
a transaction XidImpl (574641522 bq:0.0.0.0.0.0.0.0.0.0.-1.-1.10.20.103.76.-72.-12.91.64.83.113.-113.22.0.0.126.113.0.0.0.0.0.0.0.0 formatID:131077 gtxid:0.0.0.0.0.0.0.0.0.0.-1.-1.10.20.103.76.-72.-12.91.64.83.11
3.-113.22.0.0.126.111.49 base64:AAAAAAAAAAAAAP__ChRnTLj0W0BTcY8WAAB-cQAAAAAAAAAAAAAAAAAAAAAAAP__ChRnTLj0W0BTcY8WAAB-bzEHAgIA code:-6
13:50:48,113 WARN  [com.arjuna.ats.jta] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue]) ARJUNA016061: TransactionImple.enlistResource - XAResource.start returned: XAException.X
AER_PROTO for < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a14674c:-470ba4c0:53718f16:4316e, node_name=1, branch_uid=0:ffff0a14674c:-470ba4c0:53718f16:43170, subordinatenodename=null, eis_na
me=unknown eis name >: javax.transaction.xa.XAException
        at org.hornetq.core.client.impl.ClientSessionImpl.start(ClientSessionImpl.java:1775) [hornetq-core-client.jar:]
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:636)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:397)
        at org.apache.activemq.jms.pool.XaConnectionPool.createSession(XaConnectionPool.java:70) [activemq-jms-pool.jar:5.9.0]
        at org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167) [activemq-jms-pool.jar:5.9.0]
        at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196) [spring-jms.jar:3.2.2.RELEASE]
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.access$200(AbstractPollingMessageListenerContainer.java:77) [spring-jms.jar:3.2.2.RELEASE]
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer$MessageListenerContainerResourceFactory.createSession(AbstractPollingMessageListenerContainer.java:531) [spring-jms.jar:3.2.2.RE
LEASE]
        at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:300) [spring-jms.jar:3.2.2.RELEASE]
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:288) [spring-jms.jar:3.2.2.RELEASE]
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243) [spring-jms.jar:3.2.2.RELEASE]
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1096) [spring-jms.jar:3.2.2.RELEASE]
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1088) [spring-jms.jar:3.2.2.RELEASE]
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:985) [spring-jms.jar:3.2.2.RELEASE]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]


13:50:48,114 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue]) javax.transaction.xa.XAException
13:50:48,114 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.hornetq.core.client.impl.ClientSessionImpl.start(ClientSessionImpl.java:1775)
13:50:48,114 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(Transa
ctionImple.java:636)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(Transa
ctionImple.java:397)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.apache.activemq.jms.pool.XaConnectionPool.createSession(XaConnectionPool.java:70)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.access$200(Abstr
actPollingMessageListenerContainer.java:77)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.AbstractPollingMessageListenerContainer$MessageListenerC
ontainerResourceFactory.createSession(AbstractPollingMessageListenerContainer.java:531)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(Conne
ctionFactoryUtils.java:300)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExec
ute(AbstractPollingMessageListenerContainer.java:288)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecut
e(AbstractPollingMessageListenerContainer.java:243)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvo
ker.invokeListener(DefaultMessageListenerContainer.java:1096)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvo
ker.executeOngoingLoop(DefaultMessageListenerContainer.java:1088)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvo
ker.run(DefaultMessageListenerContainer.java:985)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 - JmsConsumer[asynchronousPaymentProcessingQueue])         at java.lang.Thread.run(Thread.java:662)
13:50:50,165 WARN  [org.apache.camel.component.jms.DefaultJmsMessageListenerContainer] (Camel (camelContext) thread #20 - JmsConsumer[asynchronousPaymentProcessingQueue]) Setup of JMS message listener invoker failed for destination 'asynchronousPaymentProcessingQueue' - trying to recover. Cause: JTA transaction unexpectedly rolled back (maybe due to a timeout); nested exception is javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.

13:21:38,295 ERROR [org.hornetq.core.server] (Old I/O server worker (parentId: -1887525330, [id: 0x8f7ea62e, /10.20.103.88:5595])) HQ224029: Caught XA exception: org.hornetq.core.exception.HornetQXAException: Invalid transaction state
        at org.hornetq.core.server.impl.ServerSessionImpl.individualAcknowledge(ServerSessionImpl.java:634) [hornetq-server-2.3.0.CR1.jar:]
        at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:415) [hornetq-server-2.3.0.CR1.jar:]
        at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:616) [hornetq-core-client-2.3.0.CR1.jar:]
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client-2.3.0.CR1.jar:]
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client-2.3.0.CR1.jar:]
        at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:562) [hornetq-server-2.3.0.CR1.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client-2.3.0.CR1.jar:]
        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty-3.6.2.Final.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client-2.3.0.CR1.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client-2.3.0.CR1.jar:]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty-3.6.2.Final.jar:]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) [rt.jar:1.6.0_30]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.6.0_30]
        at java.lang.Thread.run(Thread.java:701) [rt.jar:1.6.0_30]


13:21:38,354 ERROR [org.hornetq.core.server] (Old I/O server worker (parentId: -1887525330, [id: 0x8f7ea62e, /10.20.103.88:5595])) HQ224030: Caught exception: HornetQException[errorType=ILLEGAL_STATE message=HQ119057: Could not find reference on consumer ID=67, messageId = 51,567,696,761 queue = jms.queue.asynchronousPaymentProcessingQueue]
        at org.hornetq.core.server.impl.ServerConsumerImpl.acknowledge(ServerConsumerImpl.java:645) [hornetq-server-2.3.0.CR1.jar:]
        at org.hornetq.core.server.impl.ServerSessionImpl.acknowledge(ServerSessionImpl.java:624) [hornetq-server-2.3.0.CR1.jar:]
        at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:269) [hornetq-server-2.3.0.CR1.jar:]
        at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:616) [hornetq-core-client-2.3.0.CR1.jar:]
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client-2.3.0.CR1.jar:]
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client-2.3.0.CR1.jar:]
        at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:562) [hornetq-server-2.3.0.CR1.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client-2.3.0.CR1.jar:]
        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty-3.6.2.Final.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client-2.3.0.CR1.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client-2.3.0.CR1.jar:]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.2.Final.jar:]
        at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty-3.6.2.Final.jar:]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) [rt.jar:1.6.0_30]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.6.0_30]
        at java.lang.Thread.run(Thread.java:701) [rt.jar:1.6.0_30]