Adding some more logs:
This makes things a little more clear to me. So this some what explains I was seeing. This is an XA transaction and the DB part is committed but the JMS part fails. So the msg is in the db but we also get a dlq msg as the transaction failed. The msg might be redelivered and as its already in the db goes to the dlq again.
So it explains to some extent that the msg is in db and we also have 1 or more dlq.
But what could be the cause of the exception a network glitch? So when the 2nd part of the 2 phase commit happens the queue is not found?
2015-06-02 12:06:47,378 WARN [com.arjuna.ats.jta] (pool-1-thread-29) ARJUNA-16041 prepare on < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff0a28d058:126a:555566dc:23cd9c, node_name=1, branch_uid=0:ffff0a28d058:126a:555566dc:23cd9d, eis_name=unknown eis name > (com.tibco.tibjms.TibjmsXAResource@53a4e0a8) failed with exception XAException.XAER_RMERR: javax.transaction.xa.XAException
at com.tibco.tibjms.TibjmsXAResource.buildXAException(TibjmsXAResource.java:596) [:5.1.5]
at com.tibco.tibjms.TibjmsXAResource.prepareTx(TibjmsXAResource.java:757) [:5.1.5]
at com.tibco.tibjms.TibjmsXAResource.prepare(TibjmsXAResource.java:307) [:5.1.5]
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelPrepare(XAResourceRecord.java:204) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2567) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2524) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.BasicAction.prepare(BasicAction.java:2065) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1458) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:99) [:6.0.0.Final]
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:159) [:6.0.0.Final]
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1158) [:6.0.0.Final]
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119) [:6.0.0.Final]
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) [:6.0.0.Final]
at org.jboss.resource.adapter.jms.inflow.JmsServerSession$XATransactionDemarcationStrategy.end(JmsServerSession.java:657) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:243) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]
at org.jboss.resource.work.WorkWrapper.run(WorkWrapper.java:172) [:6.0.0.Final]
at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33) [:2.0.0.CR7]
at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801) [:2.0.0.CR7]
at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45) [:2.0.0.CR7]
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:842) [:2.0.0.CR7]
at java.lang.Thread.run(Thread.java:662) [:1.6.0_29]
at org.jboss.threads.JBossThread.run(JBossThread.java:122) [:2.0.0.CR7]
2015-06-02 12:06:47,378 WARN [com.arjuna.ats.arjuna] (pool-1-thread-29) ARJUNA-12073 BasicAction.End() - prepare phase of action-id 0:ffff0a28d058:126a:555566dc:23cd9c failed.
2015-06-02 12:06:47,378 WARN [com.arjuna.ats.arjuna] (pool-1-thread-29) ARJUNA-12075 Action Aborting
2015-06-02 12:06:47,378 WARN [com.arjuna.ats.jta] (pool-1-thread-29) ARJUNA-16031 XAOnePhaseResource.rollback for < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff0a28d058:126a:555566dc:23cd9c, node_name=1, branch_uid=0:ffff0a28d058:126a:555566dc:23cda9, eis_name=unknown eis name > failed with exception: org.jboss.resource.connectionmanager.JBossLocalXAException: wrong xid in rollback: expected: null, got: < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff0a28d058:126a:555566dc:23cd9c, node_name=1, branch_uid=0:ffff0a28d058:126a:555566dc:23cda9, eis_name=unknown eis name >
at org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource.rollback(TxConnectionManager.java:1168) [:6.0.0.Final]
at com.arjuna.ats.internal.jta.resources.arjunacore.XAOnePhaseResource.rollback(XAOnePhaseResource.java:186) [:6.0.0.Final]
at com.arjuna.ats.internal.arjuna.abstractrecords.LastResourceRecord.topLevelAbort(LastResourceRecord.java:123) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2902) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2881) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Abort(BasicAction.java:1883) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1471) [:6.0.0.Final]
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:99) [:6.0.0.Final]
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:159) [:6.0.0.Final]
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1158) [:6.0.0.Final]
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119) [:6.0.0.Final]
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) [:6.0.0.Final]
at org.jboss.resource.adapter.jms.inflow.JmsServerSession$XATransactionDemarcationStrategy.end(JmsServerSession.java:657) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:243) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]
at org.jboss.resource.work.WorkWrapper.run(WorkWrapper.java:172) [:6.0.0.Final]
at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33) [:2.0.0.CR7]
at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801) [:2.0.0.CR7]
at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45) [:2.0.0.CR7]
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:842) [:2.0.0.CR7]
at java.lang.Thread.run(Thread.java:662) [:1.6.0_29]
at org.jboss.threads.JBossThread.run(JBossThread.java:122) [:2.0.0.CR7]
2015-06-02 12:06:47,378 WARN [com.arjuna.ats.arjuna] (pool-1-thread-29) ARJUNA-12091 Top-level abort of action 0:ffff0a28d058:126a:555566dc:23cd9c received TwoPhaseOutcome.FINISH_ERROR from com.arjuna.ats.internal.arjuna.abstractrecords.LastResourceRecord
2015-06-02 12:06:47,394 ERROR [org.jboss.resource.adapter.jms.inflow.JmsServerSession] (pool-1-thread-29) org.jboss.resource.adapter.jms.inflow.JmsServerSession@4b1d16fb failed to commit/rollback: javax.transaction.RollbackException: ARJUNA-16053 Could not commit transaction.
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1170) [:6.0.0.Final]
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119) [:6.0.0.Final]
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) [:6.0.0.Final]
at org.jboss.resource.adapter.jms.inflow.JmsServerSession$XATransactionDemarcationStrategy.end(JmsServerSession.java:657) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:243) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]
at org.jboss.resource.work.WorkWrapper.run(WorkWrapper.java:172) [:6.0.0.Final]
at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33) [:2.0.0.CR7]
at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801) [:2.0.0.CR7]
at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45) [:2.0.0.CR7]
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:842) [:2.0.0.CR7]
at java.lang.Thread.run(Thread.java:662) [:1.6.0_29]
at org.jboss.threads.JBossThread.run(JBossThread.java:122) [:2.0.0.CR7]
2015-06-02 12:06:47,394 ERROR [org.jboss.resource.adapter.jms.inflow.JmsServerSession] (pool-1-thread-29) org.jboss.resource.adapter.jms.inflow.JmsServerSession@4b1d16fb failed to commit/rollback: javax.transaction.RollbackException: ARJUNA-16053 Could not commit transaction.
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1170) [:6.0.0.Final]
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119) [:6.0.0.Final]
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) [:6.0.0.Final]
at org.jboss.resource.adapter.jms.inflow.JmsServerSession$XATransactionDemarcationStrategy.end(JmsServerSession.java:657) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:243) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]
at org.jboss.resource.work.WorkWrapper.run(WorkWrapper.java:172) [:6.0.0.Final]
at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33) [:2.0.0.CR7]
at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801) [:2.0.0.CR7]
at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45) [:2.0.0.CR7]
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:842) [:2.0.0.CR7]
at java.lang.Thread.run(Thread.java:662) [:1.6.0_29]
at org.jboss.threads.JBossThread.run(JBossThread.java:122) [:2.0.0.CR7]