1 2 Previous Next 15 Replies Latest reply on Oct 22, 2008 8:36 AM by marklittle

    Warning after non-serializable XAResource recovery test

    szhigunov

      Hi,

      I am testing local JTA TM in JBoss. My transaction includes one serializable and one non-serializable XAResource (recovery module is configured).

      Scenario one (no issues). Kill server after prepare phase + participants are recorded in TM Object Store. Start the server. Recovery happens. Records are removed from Object Store. No warnings.

      Scenario two (possible problem). Kill server after: prepare phase + participants are recorded in TM Object Store + participants committed successfully, but before the record was removed from Object Store. Start the server. Keep getting "Could not find new XAResource to use for recovering non-serializable" warning until clean the Object Store manually.

      If this warning is an expected behavior in this case and manual clean-up required, or there is a problem with the configuration / participant?

      I can see in the debugger the recovery module is contacted and returns XAResource which can not find any transactions (as there are non).

      Thanks.


        • 1. Re: Warning after non-serializable XAResource recovery test
          szhigunov

          When I used term "recovery module" I actually meant "XAResourceRecovery implementation". Sorry for the error.

          • 2. Re: Warning after non-serializable XAResource recovery test
            adinn

             


            If this warning is an expected behavior in this case and manual clean-up required, or there is a problem with the configuration / participant?

            I can see in the debugger the recovery module is contacted and returns XAResource which can not find any transactions (as there are non).


            Yes, this is the expected behaviour. The transaction has finished so none of the recovery modules (including the XA module) finds the resource. The recovery process logs this outcome using the norecoveryxa error message.

            If you think about it you will see that this is also what would happen if the transaction was still active but you failed to install the XA recovery module. None of the recovery modules would return a resource in this case so the norecoveryxa error message gets logged. The two cases are indistinguishable to the transaction manager because it does not (cannot) know which resources still exist and which have been removed.

            • 3. Re: Warning after non-serializable XAResource recovery test
              marklittle

              Plus if you leave it for a while the recovery manager will eventually move records it cannot recover on the off chance that it did recover them but the response was lost. You can then clean them up (or not). We always fail safe.

              • 4. Re: Warning after non-serializable XAResource recovery test
                jeremystone

                Co-incidentally, I have just been looking into exactly the same issue that we have been seeing for some time. Recovery was being attempted but the xa resource recover() method was returning no xids. I had just come to the same conclusion - i.e. that the resources were committing before the object store records could be removed.

                Mark: Where does the recovery manager move the records eventually (we're using the JDBC object store) and how is this 'eventually' period configured? I can then code up something that will remove these records automatically (or is there already something available?)

                (I wouldn't be surprised if there are many people out there who see the message and assume that XA recovery is not working. Maybe something should be added to your Why do I get "Could not find new XAResource to use for recovering non-serializable XAResource" (http://www.jboss.org/community/docs/DOC-12826) Wiki page???)

                • 5. Re: Warning after non-serializable XAResource recovery test
                  adinn

                   


                  Co-incidentally, I have just been looking into exactly the same issue that we have been seeing for some time. Recovery was being attempted but the xa resource recover() method was returning no xids. I had just come to the same conclusion - i.e. that the resources were committing before the object store records could be removed.


                  Hmm, that would be quite a coincidence. The chances of this happening without any intervention are somewhere between slim and none. It requires the coordinator thread to successfully communicate a prepare and a commit to all participants then die while waiting for the final participant to acknowledge the last commit (ok, it might, at some yet more incomputable level of improbability, die after receiving the commit and before deleting the log record). If this is happening regularly then it probably indicates that something is very wrong -- most probably that the communication with the last participant is killing either the coordinator thread or the JVM. Possibly, the object store disk might be flaky. Anyway, you are very unlikely to see this unless you have a flaky resource/platform or you engineer the situation in a debugger or some other test rig.

                  Any comment in the Wiki page needs the above qualification in BOLD CAPS

                  • 6. Re: Warning after non-serializable XAResource recovery test
                    jeremystone

                    To be clear: we see this while testing after we kill the VM. It is not happening spontaneousy. But we don't have to try too hard to get it to happen.

                    • 7. Re: Warning after non-serializable XAResource recovery test
                      marklittle

                       


                      Mark: Where does the recovery manager move the records eventually (we're using the JDBC object store) and how is this 'eventually' period configured? I can then code up something that will remove these records automatically (or is there already something available?)


                      This should be in the Failure Recovery guide. Please check and let me know if you can't find it.

                      • 8. Re: Warning after non-serializable XAResource recovery test
                        jeremystone

                        Can find reference to ExpiryScanner of which the only implementation in 4.2.3.sp7 shipped with JBossAS 4.2.3 seems to be the ExpiredTransactionStatusManagerScanner. (This looks like it removes old /Recovery/TransactionStatusManager records - which is not quite what we want, I guess).

                        I notice from SVN that ExpiredTransactionScanner and AtomicActionExpiryScanner were added for JBTM-300/JBTM-301. Are these what you refer to? (Looks like they're scheduled for 4.4.0.ga - as yet unreleased - will will these classes work against 4.2.3.sp7, do you think, if I grab the source?)

                        • 9. Re: Warning after non-serializable XAResource recovery test
                          szhigunov

                          To confirm what was said above:
                          1) To reproduce this scenario I stepped through TM code in debugger and killed the server when it was about to update Object Store.
                          2) I use JBoss 4.2.2, it comes with the only one expiry scanner: ExpiredTransactionStatusManagerScanner. Just in case, I waited more then 12 hours (period) and the warning did not go anywhere.

                          • 10. Re: Warning after non-serializable XAResource recovery test
                            marklittle

                            Hi Jeremy. I'll take a look at the docs and get back to you.

                            • 11. Re: Warning after non-serializable XAResource recovery test
                              jeremystone

                              Thanks.

                              Have tried using ExpiredTransactionScanner with v 4.2.3. It seems to insert a new /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction/Expired record for each old /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction record. The /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction do not get deleted.

                              Is there something else that will do the deletion or have I misunderstood what it does?

                              • 12. Re: Warning after non-serializable XAResource recovery test
                                jeremystone

                                To get it to delete the original log entry (with the JDBC object store that we are using) and get the recovery warning to (eventually) go away we had to modify the ExpiredTransactionScanner.scan() method to add:

                                 if (moved)
                                 {
                                 moved = _objectStore
                                 .remove_committed(newUid,
                                 _typeName);
                                 }
                                


                                immediately after the call to _objectStore.write_committed(...).

                                • 13. Re: Warning after non-serializable XAResource recovery test
                                  marklittle

                                  Hi Jeremy. Yes, you're right and the fix is correct.

                                  I've created https://jira.jboss.org/jira/browse/JBTM-418

                                  • 14. Re: Warning after non-serializable XAResource recovery test
                                    jeremystone

                                    Glad to be of help ;-)

                                    1 2 Previous Next