2 Replies Latest reply on Jun 5, 2015 12:32 PM by rahul2015

    TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch errors in Prod

    rahul2015

      Hi All,

      I have been facing an issue in production which I need some help on. We have a cluster JBoss (Jboss6.0.0.Final) which reads of messages of Tibco EMS queue (Tibco8.x).

      The Tibco is configured for Windows failover.

      Looks like the Tibco is failing over and we have the exceptions seen in the JBoss logs

       

       

      2015-06-02 12:06:19,704 ERROR [org.jboss.resource.adapter.jms.inflow.JmsServerSession] (pool-1-thread-72) Failed to commit session transaction: javax.jms. !TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back

                     at com.tibco.tibjms.Tibjmsx.buildException(Tibjmsx.java:582) [:5.1.5]

                     at com.tibco.tibjms.TibjmsxSessionImp._confirmTransacted(TibjmsxSessionImp.java:3107) [:5.1.5]

                     at com.tibco.tibjms.TibjmsxSessionImp._confirm(TibjmsxSessionImp.java:3502) [:5.1.5]

                     at com.tibco.tibjms.TibjmsxSessionImp._commit(TibjmsxSessionImp.java:2766) [:5.1.5]

                     at com.tibco.tibjms.TibjmsxSessionImp.commit(TibjmsxSessionImp.java:4692) [:5.1.5]

                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession$LocalDemarcationStrategy.end(JmsServerSession.java:485) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]

                     at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:243) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]

                     at org.jboss.resource.work.WorkWrapper.run(WorkWrapper.java:172) [:6.0.0.Final]

                     at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33) [:2.0.0.CR7]

                     at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801) [:2.0.0.CR7]

                     at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45) [:2.0.0.CR7]

                     at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:842) [:2.0.0.CR7]

                     at java.lang.Thread.run(Thread.java:662) [:1.6.0_29]

                     at org.jboss.threads.JBossThread.run(JBossThread.java:122) [:2.0.0.CR7]

                    

      But what confuses me is:

      1. Why does this happen?
      2. The msgs are persistent and some time they are redelivered.
      3. Sometimes they are not redelivered.
      4. Sometimes I see them directly in DLQ
      5. Sometimes they are in the database ie our transaction is committed but they are still redelivered

       

      If the msgs are persistent and not acknowledged they will be redelivered that might be due to the Tibco flipping over. But this can’t explain all the above scenario’s I see.

       

      Can someone shed some light.

       

      Thanks

      Rahul

        • 1. Re: TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch errors in Prod
          rahul2015

          Adding some more logs:

           

          This makes things a little more clear to me. So this some what explains I was seeing. This is an XA transaction and the DB part is committed but the JMS part fails. So the msg is in the db but we also get a dlq msg as the transaction failed. The msg might be redelivered and as its already in the db goes to the dlq again.

          So it explains to some extent that the msg is in db and we also have 1 or more dlq.

           

          But what could be the cause of the exception a network glitch? So when the 2nd part of the 2 phase commit happens the queue is not found?

           

           

          2015-06-02 12:06:47,378 WARN  [com.arjuna.ats.jta] (pool-1-thread-29) ARJUNA-16041 prepare on < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff0a28d058:126a:555566dc:23cd9c, node_name=1, branch_uid=0:ffff0a28d058:126a:555566dc:23cd9d, eis_name=unknown eis name > (com.tibco.tibjms.TibjmsXAResource@53a4e0a8) failed with exception XAException.XAER_RMERR: javax.transaction.xa.XAException

            at com.tibco.tibjms.TibjmsXAResource.buildXAException(TibjmsXAResource.java:596) [:5.1.5]

            at com.tibco.tibjms.TibjmsXAResource.prepareTx(TibjmsXAResource.java:757) [:5.1.5]

            at com.tibco.tibjms.TibjmsXAResource.prepare(TibjmsXAResource.java:307) [:5.1.5]

            at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelPrepare(XAResourceRecord.java:204) [:6.0.0.Final]

            at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2567) [:6.0.0.Final]

            at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2524) [:6.0.0.Final]

            at com.arjuna.ats.arjuna.coordinator.BasicAction.prepare(BasicAction.java:2065) [:6.0.0.Final]

            at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1458) [:6.0.0.Final]

            at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:99) [:6.0.0.Final]

            at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:159) [:6.0.0.Final]

            at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1158) [:6.0.0.Final]

            at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119) [:6.0.0.Final]

            at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) [:6.0.0.Final]

            at org.jboss.resource.adapter.jms.inflow.JmsServerSession$XATransactionDemarcationStrategy.end(JmsServerSession.java:657) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]

            at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:243) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]

            at org.jboss.resource.work.WorkWrapper.run(WorkWrapper.java:172) [:6.0.0.Final]

            at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33) [:2.0.0.CR7]

            at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801) [:2.0.0.CR7]

            at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45) [:2.0.0.CR7]

            at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:842) [:2.0.0.CR7]

            at java.lang.Thread.run(Thread.java:662) [:1.6.0_29]

            at org.jboss.threads.JBossThread.run(JBossThread.java:122) [:2.0.0.CR7]

           

           

          2015-06-02 12:06:47,378 WARN  [com.arjuna.ats.arjuna] (pool-1-thread-29) ARJUNA-12073 BasicAction.End() - prepare phase of action-id 0:ffff0a28d058:126a:555566dc:23cd9c failed.

          2015-06-02 12:06:47,378 WARN  [com.arjuna.ats.arjuna] (pool-1-thread-29) ARJUNA-12075 Action Aborting

          2015-06-02 12:06:47,378 WARN  [com.arjuna.ats.jta] (pool-1-thread-29) ARJUNA-16031 XAOnePhaseResource.rollback for < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff0a28d058:126a:555566dc:23cd9c, node_name=1, branch_uid=0:ffff0a28d058:126a:555566dc:23cda9, eis_name=unknown eis name > failed with exception: org.jboss.resource.connectionmanager.JBossLocalXAException: wrong xid in rollback: expected: null, got: < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff0a28d058:126a:555566dc:23cd9c, node_name=1, branch_uid=0:ffff0a28d058:126a:555566dc:23cda9, eis_name=unknown eis name >

            at org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource.rollback(TxConnectionManager.java:1168) [:6.0.0.Final]

            at com.arjuna.ats.internal.jta.resources.arjunacore.XAOnePhaseResource.rollback(XAOnePhaseResource.java:186) [:6.0.0.Final]

            at com.arjuna.ats.internal.arjuna.abstractrecords.LastResourceRecord.topLevelAbort(LastResourceRecord.java:123) [:6.0.0.Final]

            at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2902) [:6.0.0.Final]

            at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2881) [:6.0.0.Final]

            at com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Abort(BasicAction.java:1883) [:6.0.0.Final]

            at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1471) [:6.0.0.Final]

            at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:99) [:6.0.0.Final]

            at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:159) [:6.0.0.Final]

            at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1158) [:6.0.0.Final]

            at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119) [:6.0.0.Final]

            at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) [:6.0.0.Final]

            at org.jboss.resource.adapter.jms.inflow.JmsServerSession$XATransactionDemarcationStrategy.end(JmsServerSession.java:657) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]

            at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:243) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]

            at org.jboss.resource.work.WorkWrapper.run(WorkWrapper.java:172) [:6.0.0.Final]

            at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33) [:2.0.0.CR7]

            at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801) [:2.0.0.CR7]

            at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45) [:2.0.0.CR7]

            at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:842) [:2.0.0.CR7]

            at java.lang.Thread.run(Thread.java:662) [:1.6.0_29]

            at org.jboss.threads.JBossThread.run(JBossThread.java:122) [:2.0.0.CR7]

           

           

          2015-06-02 12:06:47,378 WARN  [com.arjuna.ats.arjuna] (pool-1-thread-29) ARJUNA-12091 Top-level abort of action 0:ffff0a28d058:126a:555566dc:23cd9c received TwoPhaseOutcome.FINISH_ERROR from com.arjuna.ats.internal.arjuna.abstractrecords.LastResourceRecord

           

           

           

           

          2015-06-02 12:06:47,394 ERROR [org.jboss.resource.adapter.jms.inflow.JmsServerSession] (pool-1-thread-29) org.jboss.resource.adapter.jms.inflow.JmsServerSession@4b1d16fb failed to commit/rollback: javax.transaction.RollbackException: ARJUNA-16053 Could not commit transaction.

            at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1170) [:6.0.0.Final]

            at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119) [:6.0.0.Final]

            at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) [:6.0.0.Final]

            at org.jboss.resource.adapter.jms.inflow.JmsServerSession$XATransactionDemarcationStrategy.end(JmsServerSession.java:657) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]

            at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:243) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]

            at org.jboss.resource.work.WorkWrapper.run(WorkWrapper.java:172) [:6.0.0.Final]

            at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33) [:2.0.0.CR7]

            at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801) [:2.0.0.CR7]

            at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45) [:2.0.0.CR7]

            at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:842) [:2.0.0.CR7]

            at java.lang.Thread.run(Thread.java:662) [:1.6.0_29]

            at org.jboss.threads.JBossThread.run(JBossThread.java:122) [:2.0.0.CR7]

           

           

           

           

          2015-06-02 12:06:47,394 ERROR [org.jboss.resource.adapter.jms.inflow.JmsServerSession] (pool-1-thread-29) org.jboss.resource.adapter.jms.inflow.JmsServerSession@4b1d16fb failed to commit/rollback: javax.transaction.RollbackException: ARJUNA-16053 Could not commit transaction.

            at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1170) [:6.0.0.Final]

            at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:119) [:6.0.0.Final]

            at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75) [:6.0.0.Final]

            at org.jboss.resource.adapter.jms.inflow.JmsServerSession$XATransactionDemarcationStrategy.end(JmsServerSession.java:657) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]

            at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:243) [:5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)]

            at org.jboss.resource.work.WorkWrapper.run(WorkWrapper.java:172) [:6.0.0.Final]

            at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33) [:2.0.0.CR7]

            at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801) [:2.0.0.CR7]

            at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45) [:2.0.0.CR7]

            at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:842) [:2.0.0.CR7]

            at java.lang.Thread.run(Thread.java:662) [:1.6.0_29]

            at org.jboss.threads.JBossThread.run(JBossThread.java:122) [:2.0.0.CR7]

          • 2. Re: TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch errors in Prod
            rahul2015

            Can anyone explain looking at the logs what is happening?