9 Replies Latest reply on Dec 4, 2008 2:39 AM by Mark Little

    how to test recovery?

    mazz Master

      Suppose I have fully configured recovery in my JBossAS 4.2.1 using the instructions found here: http://www.jboss.org/community/docs/DOC-10789.

      I have a J2EE/EJB3 app deployed that uses JMS messaging and my own XA datasource. I am using the AppServerJDBCXARecovery object mentioned in that wiki.

      What can I do to trigger a transaction failure such that I can test recovery? What kind of test code can I write to simulate a recoverable failure?

      I saw some example code on that wiki page, but it looks like it is for testing custom XAResources (the wiki states, "For recovery of custom XAResource types not covered in the sections above, keep reading." but I'm recovering FOR a XAResource type covered above - specifically XA datasources whose XAResources are found in the oracle/postgres JDBC drivers and that are obtained by the AppServerJDBCXARecovery object).

      I'm not quite sure what exactly has to happen for recovery to take place. Just causing a transaction to rollback under "normal circumstances" isn't going to do it, I don't think. Catastrophes such as a database crashing or a network going down would cause it I would think, but I'm not sure how to "simulate" a transaction failure (via code) that would cause recover to happen.

        • 1. Re: how to test recovery?
          Andrew Dinn Master

          One way to force recovery to kick in is to cause the JVM to crash between prepare and commit. At the moment the only easy way to do this is to use a debugger to stop the TX in mid flight and then kill the JVM process. Of course this sort of procedure is far from automatic and unable to deal with anything beyond the most basic scenarios.

          Now, strangely enough, it just so happens that I have recently developed a testing tool to address precisely this issue. It uses a rule based scripting language to introduce side effects into method code. These side effects can be arbitrary bits of Java code, including throws of synthetic exceptions, early return, possibly with a synthetic result value, and various thread delay, synchronization and sequencing operations which allow repeatable behaviour from non-deterministic code.

          I have been trying this tool out testing recovery in the the Web Services Transactions code with great success. I have not yet made the code or documentation publicly accessible but watch for an announcement in the near future. It should be available soon under the GPL. If you want to be a guinea pig I'll be glad to provide best-effort support.

          regards,


          Andrew Dinn
          JBoss Web Services Transactions Lead

          • 2. Re: how to test recovery?
            mazz Master

            Andrew - thanks. I'll be interested in trying it out whenever its ready.

            • 3. Re: how to test recovery?
              Vicky Kak Master

               

              "adinn" wrote:
              One way to force recovery to kick in is to cause the JVM to crash between prepare and commit. At the moment the only easy way to do this is to use a debugger to stop the TX in mid flight and then kill the JVM process. Of course this sort of procedure is far from automatic and unable to deal with anything beyond the most basic scenarios.

              How about adding another dummy XAResource in the ongoing TX, the problem will be the order in which the participants would be called?
              Ofcource this dummy XAResource/participant would be halting the TX.

              • 4. Re: how to test recovery?
                Andrew Dinn Master

                Yes, you can use a dummy resource. This is not quite so straightforward as stopping the code in a debugger because it requires implementing and registering the faulting resource. Also, registering another resource changes the scenario which has consequences for how things execute. So, for example, you could not test how your resource copes with JVM crashes during one phase commit using this method since a one phase commit will never happen. The ordering issue also means that you cannot guarantee whether the real resource has been prepared or not before the dummy one gets to kill the JVM. My testing tool allows these issues to be avoided more gracefully.

                • 5. Re: how to test recovery?
                  Vicky Kak Master

                   

                  "adinn" wrote:
                  My testing tool allows these issues to be avoided more gracefully.

                  Yes I understand the drawback of dummy participant.
                  Let me have a look at your testing tool which looks interesting to me.

                  • 6. Re: how to test recovery?
                    Andrew Dinn Master

                    Mazz,

                    If you want to try using my test tool then please download it from the JBossTS repository at the following URL.

                    http://anonsvn.jboss.org/repos/labs/labs/jbosstm/workspace/adinn/orchestration/

                    The README gives instructions fo rusing it in brief. You will find a more full manual in the docs subdirectory.

                    regards,


                    Andrew Dinn

                    • 7. Re: how to test recovery?
                      Mark Little Master

                       

                      "adinn" wrote:
                      Yes, you can use a dummy resource. This is not quite so straightforward as stopping the code in a debugger because it requires implementing and registering the faulting resource. Also, registering another resource changes the scenario which has consequences for how things execute. So, for example, you could not test how your resource copes with JVM crashes during one phase commit using this method since a one phase commit will never happen. The ordering issue also means that you cannot guarantee whether the real resource has been prepared or not before the dummy one gets to kill the JVM. My testing tool allows these issues to be avoided more gracefully.


                      Let's not forget that you can enforce ordering within the intentions list if you go beyond the standards. After all, that's what our QA suite does in order to test crash failures and recovery automatically.

                      • 8. Re: how to test recovery?
                        Vicky Kak Master

                         

                        "mark.little@jboss.com" wrote:

                        Let's not forget that you can enforce ordering within the intentions list if you go beyond the standards.

                        I am curious to find out how this is done?
                        Can you point out me the code from svn if it exists there?

                        • 9. Re: how to test recovery?
                          Mark Little Master

                          As Obi Wan Kenobi once said (or would have if we were a software engineer): "Looks in the manuals my padawan." ;-)