6 Replies Latest reply on Sep 8, 2014 2:44 PM by tbronzan

    Periodic Recovery - ARJUNA016037 and ARJUNA016038

    tbronzan

      Occasionally when WildFly is topped and started again I get a bunch of the following warnings in the log file:

       

      2014-08-18 05:15:08,787 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffc0a815a7:-7aef0ab8:53dbd0f0:b02, node_name=1, branch_uid=0:ffffc0a815a7:-7aef0ab8:53dbd0f0:b06, subordinatenodename=null, eis_name=java:jboss/jdbc/realtime >, heuristic: TwoPhaseOutcome.FINISH_OK, product: H2/1.3.173 (2013-07-28), jndiName: java:jboss/jdbc/realtime com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@7955927d >
      
      
      2014-08-18 05:15:08,787 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffc0a815a7:-7aef0ab8:53dbd0f0:b02, node_name=1, branch_uid=0:ffffc0a815a7:-7aef0ab8:53dbd0f0:b06, subordinatenodename=null, eis_name=java:jboss/jdbc/realtime >
      

       

      From what I understand it's from transactions that didn't complete when the server was shut down and it's trying to resume them when it starts up again.  I'm looking for the proper way to get rid of these messages.  Everything that I've come across so far has said to just delete the files under the data/tx-object-store folder, but that's not really a great solution.  Due to the nature of our application it's safe for us to discard those transactions rather than try to commit them.  Is there a better way to deal with this than deleting the files?

        • 1. Re: Periodic Recovery - ARJUNA016037 and ARJUNA016038
          ochaloup

          Hi,

           

          I think that this issue was discussed several times in other threads. I would recommend to check explanation here: norecoveryxa | Planet JBoss Developer

          Or other reason for this warning could be that there are some unfinished transactions in jboss transaction log bound to datasource that the app server does not know. I mean something like that configuration of standalone.xml does not contain datasource with jndi "java:jboss/jdbc/realtime". But it does not seem being the case here.

           

          I would say that the best solution is use jboss cli. It could be started by $JBOSS_HOME/bin/jboss-cli.sh from command line.

          First you need to "load" data from the transaction log by probing log-store

          /subsystem=transactions/log-store=log-store:probe()

          Then you can see which transactions are unfinished there:

          /subsystem=transactions/log-store=log-store:read-resource(recursive=true)

          And you can delete such transaction one by one from the log store. This means that they will be forgotten. There will be no rollback called on any resource that participated in transaction. But there will be no in-doubt transaction in log so transaction manager will not complain about them anymore.

          /subsystem=transactions/log-store=log-store/transactions=0\:ffffc0a815a7\:-7aef0ab8\:53dbd0f0\:b02:delete()

           

          Ondra

          1 of 1 people found this helpful
          • 2. Re: Periodic Recovery - ARJUNA016037 and ARJUNA016038
            tbronzan

            Thanks for the response.  Unfortunately using jboss cli is not an option because we need something that will happen automatically.  We can't expect a customer to know how WIldFly works and how to handle problems like these manually.  Furthermore, we cannot monitor thousands of customers to see if they are getting these log messages.  We are looking for a solution that will automatically discard the transactions.  I'm trying to avoid deleting the the data/tx-object-store folder on startup, but it's starting to look like I may have to.

            • 3. Re: Periodic Recovery - ARJUNA016037 and ARJUNA016038
              j_ri

              Yes, it was discussed several times, but there are no good answers;-(

               

              I tested with Wildfly, EAP 6.1.1, EAP 6.2.4 and EAP 6.3.0

               

              Although there are these entries in the log

              16:03:02,301 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=39, bqual_length=36, tx_uid=0:ffff96854d53:-362d13b0:54047c43:10, node_name=rk3983node1, branch_uid=0:ffff96854d53:-362d13b0:54047c43:1b, subordinatenodename=null, eis_name=java:jboss/datasources/XATestNode11_Informix_DS >, heuristic: TwoPhaseOutcome.FINISH_OK, product: Informix Dynamic Server/11.50.FC7, jndiName: java:jboss/datasources/XATestNode11_Informix_DS com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@136e8bfd >16:03:02,302 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=39, bqual_length=36, tx_uid=0:ffff96854d53:-362d13b0:54047c43:10, node_name=rk3983node1, branch_uid=0:ffff96854d53:-362d13b0:54047c43:29, subordinatenodename=null, eis_name=java:jboss/datasources/XATestNode21_Informix_DS >, heuristic: TwoPhaseOutcome.FINISH_OK, product: Informix Dynamic Server/11.50.FC7, jndiName: java:jboss/datasources/XATestNode21_Informix_DS com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@544017e9 >16:03:02,302 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=39, bqual_length=36, tx_uid=0:ffff96854d53:-362d13b0:54047c43:10, node_name=rk3983node1, branch_uid=0:ffff96854d53:-362d13b0:54047c43:2e, subordinatenodename=null, eis_name=java:jboss/datasources/XATestNode22_DS >, heuristic: TwoPhaseOutcome.FINISH_OK, product: Oracle/Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production, jndiName: java:jboss/datasources/XATestNode22_DS com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@3be1ed35 >16:03:02,302 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=39, bqual_length=36, tx_uid=0:ffff96854d53:-362d13b0:54047c43:10, node_name=rk3983node1, branch_uid=0:ffff96854d53:-362d13b0:54047c43:1b, subordinatenodename=null, eis_name=java:jboss/datasources/XATestNode11_Informix_DS >16:03:02,302 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=39, bqual_length=36, tx_uid=0:ffff96854d53:-362d13b0:54047c43:10, node_name=rk3983node1, branch_uid=0:ffff96854d53:-362d13b0:54047c43:29, subordinatenodename=null, eis_name=java:jboss/datasources/XATestNode21_Informix_DS >16:03:02,302 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=39, bqual_length=36, tx_uid=0:ffff96854d53:-362d13b0:54047c43:10, node_name=rk3983node1, branch_uid=0:ffff96854d53:-362d13b0:54047c43:2e, subordinatenodename=null, eis_name=java:jboss/datasources/XATestNode22_DS >

               

               

              the CLI commands

              /subsystem=transactions/log-store=log-store:probe()

              and

              /subsystem=transactions/log-store=log-store:read-resource(recursive=true)

              don't find any results.

               

               

              after deleting the "tx-object-store" folder and starting the server again, the pending transactions get rollbacked....but this should happen before....

              Why is "public void rollback(Xid xid) throws XAException" not called before (e.g. after reconnection to the database after a network crash )??

               

              Is this really the way the recovery manager should work?

               

              Regards,

              Jochen

              • 4. Re: Periodic Recovery - ARJUNA016037 and ARJUNA016038
                tomjenkinson

                Hi guys,

                 

                This should be resolved in JBTM-860.

                 

                Hope it helps,

                Tom

                • 5. Re: Periodic Recovery - ARJUNA016037 and ARJUNA016038
                  ochaloup

                  Hi Troy,

                   

                  There is property xaAssumeRecoveryComplete which assumes the recovery being completed and unknown txn log entries could be forgotten. This should work with JTA but I think that it does not work for JTS. You put it as property on start of JBoss AS/WildFly like -DJTAEnvironmentBean.xaAssumeRecoveryComplete=true

                   

                  I'm not sure if this helps and whether this would be a safe settings for you - but maybe as workaround it could work (till JBTM-860 will be implemented)

                   

                  Detailed info on the option could be found in TM recovery guide here

                  Failure Recovery Guide - Assumed complete

                   

                  Ondra

                  • 6. Re: Periodic Recovery - ARJUNA016037 and ARJUNA016038
                    tbronzan

                    Thank you for the response.  Setting that property to true seems to have resolved the issue.