8 Replies Latest reply on Jan 10, 2012 8:59 AM by jhalliday

    Arquillian transaction support

    dan.j.allen

      We are interested in gathering feedback from transaction experts, notably the TS team, about how to offer proper transaction support in Arquillian. If we are going to do transaction support, we want to do it right, not just slap on some annotations and call it done. After all, in the Arquillian team, we believe in real tests.

       

      The Arquillian Persistence extension includes initial support for transactions, thanks to the great work of Bartosz. We are looking to build on and strengthen that functionality.

       

      Here's a little background to get everyone on the same page.

       

      Community members have often requested the automatic transaction and rollback behavior that SpringTest offers. This functionality is now offered by the Arquillian Persistence extension, as handled by the TransactionWrapper class.

       

      However, we've often argued that a simple transaction rollback after the execution of the test method can lead to false positives, particularly with JPA. The reason is that defered DML statements often don't get executed due to the timing of the rollback.

       

      To be more specific, the problem is that persistence layers "save up" writes to the database until the last minute, just before the 2PC occurs. If the client marks the transaction as rollback only, or invokes UserTransaction.rollback(), the TM optimizes by not calling the beforeCompletion phase, which prevents the updates being sent to the DB. This results in issues like mapping layer miss-matches being left undetected.

       

      On the other hand, manual transaction logic in the test isn't ideal either. It's also error prone and abandons the "component model for tests" spirit of Arquillian. We want something both correct and something developers will embrace.

        • 1. Re: Arquillian transaction support
          paul.robinson

          Dan,

           

          Are you able to provide an example of a test that uses @Transactional(TransactionMode.ROLLBACK)? I'm wondering, what is the use case for this feature? I guess some people will use it to  revert the test data back to it's pre-test state. This seems like a bad idea as there are better ways to do this.

           

          I'm thinking, the real use case is:

           

          1. Write a test that makes some changes to the application state
          2. Annotate the test method with @Transactional(TransactionMode.ROLLBACK)
          3. Check the state of the system in an @AfterTest annotated method? Or if using, the persistence-extension, specify a @ShouldMatchDataSet to specify what the data should look like after the rollback.

           

          If this is the case, what are we testing here? It seems like we are testing that the middleware is performing correctly? Of course, this could be a valid test, I'm just trying to get my head around who the target audience is for this feature. I have some tests in the Narayana (rebranding of JBossTS) project that would benefit from this feature, but here we are testing the middleware, rather than an application.

           

          I've found some tests that use @Transactional(TransactionMode.ROLLBACK) in arquillian-extension-persistence. One in particular is this one:

           

          https://github.com/bartoszmajsak/arquillian-extension-persistence/blob/master/int-tests/src/test/java/org/jboss/arquillian/example/UserPersistenceTest.java

           

          'shouldPersistUsersAndRollbackTransactionAfterTestExecution' and 'shouldPersistUsersWithinTransaction' seem to have the same logic, but one is set to rollback after the test. I can't see anywhere in the test that asserts the rollback did as intended. Is this test missing something, or is it me that's missing something?

           

          Paul.

          • 2. Re: Arquillian transaction support
            paul.robinson

            Do we know of a way to assert that the DML commands have fired? It would be great to provide a test case for this issue. Also, It would also be nice to run it on Spring to see if they have already solved this issue.

             

            Paul.

            • 3. Re: Arquillian transaction support
              paul.robinson

              Dan,

               

              I think I missed something when I stated this in our original email thread:

              To be more specific, the problem is that persistence layers "save up" writes to the database until the last minute, just before the 2PC occurs. If the client marks the transaction as rollback only, or invokes UserTransaction.rollback(), the TM optimizes by not calling the beforeCompletion phase, which prevents the updates being sent to the DB. This results in issues like mapping layer miss-matches being left undetected.

               

               

              This only applies for resource managers that use synchronizations to flush the cache. My understanding is that some flush changes during the prepare phase of the 2PC protocol. It's also possible that some RMs might do some other 'magic' to flush the data at some other time. This last statement is entirely speculation; but the point is that our users will be using RMs that we have never tested, so we need to be sure that our assumptions around this are valid if we create a solution that relies on them.

               

              Paul.

              • 4. Re: Arquillian transaction support
                paul.robinson

                I have an idea that may help. However, in it's current form, it will only work for RMs that flush during the synchronization protocol.

                 

                Essentially the problem is that the beforeCompletion phase in the protocol is not being fired. We need to cause the transaction to rollback at a later stage in the protocol. We could do this by enlisting a mock participant that always votes rollback. We then have Arquillian call commit, rather than Rollback. What should happen is:

                 

                1. Arquillian registers a participant that always votes rollback on call to prepare.
                2. Arquillian invokes UserTransaction.commit() after the test method has invoked
                3. The beforeCompletion should fire for all participants.
                4. The persistence layer flushes it's changes
                5. The prepare phase begins and all participants are asked to prepare
                6. Our mock participant votes to rollback the transaction
                7. Rollback is called on all participants
                8. The client receives a RollbackException on the call to UserTransaction.commit()
                9. Arquillian checks the rollback occurred, due to our mock participant.

                 

                As a result, TransactionalWrapper#afterTest would look something like this:

                 

                   public void afterTest(@Observes EndTransaction endTransaction) throws Exception
                    {
                       ...
                       if (TransactionMode.COMMIT.equals(mode))
                       {
                          obtainTransaction().commit();
                       }
                       else
                       {
                           TransactionManager tm = (TransactionManager) contextInstance.lookup("...");
                           RoolbackParticipant p = new RollbackPrticipant();
                           tm.getTransaction().enlistResource(p);
                           try
                           {
                               obtainTransaction().commit();
                           }
                           catch (RollbackException e)
                           {
                                 //If didn't rollback due to our participant, register a test failure
                           }
                       }
                    }
                

                 

                Potential issues:

                 

                1. Is the pre-prepare phase invoked on all participants before prepare is called on any participant? Otherwise, our rollback participant may vote to rollback before the pre-prepare on the DB participant.
                2. Will only work if RM uses synchronizations to flush data.

                 

                There might be a way of ordering prepares in Narayana. This would allow us to be the last participant and thus support RMs that flush during prepare. If such a feature did exist, it would be non-standard, and unlikely to work on an AS that didn't use Narayana.

                • 5. Re: Arquillian transaction support
                  paul.robinson

                  Paul Robinson wrote:

                  To be more specific, the problem is that persistence layers "save up" writes to the database until the last minute, just before the 2PC occurs. If the client marks the transaction as rollback only, or invokes UserTransaction.rollback(), the TM optimizes by not calling the beforeCompletion phase, which prevents the updates being sent to the DB. This results in issues like mapping layer miss-matches being left undetected.

                   

                   

                  This only applies for resource managers that use synchronizations to flush the cache. My understanding is that some flush changes during the prepare phase of the 2PC protocol. It's also possible that some RMs might do some other 'magic' to flush the data at some other time. This last statement is entirely speculation; but the point is that our users will be using RMs that we have never tested, so we need to be sure that our assumptions around this are valid if we create a solution that relies on them.

                   

                   

                  A cunning trick could be to use a one-phase aware resource as our mock participant. One phase resources, are ones which are not capable, or do not wish to prepare. Only one instance of these can be enlisted in a 2PC. The one-phase resource is instructed to commit after all the other participants have prepared. If the commit succeeds, all the other resources are committed. if the commit fails, then all other participants are rolled back. This feature is typically used if you use an RM that has not implemented a correct prepare method, or for a critical resource that you do not wish to lock for the duration of the 2PC.

                   

                  I think this is the latest we can hook into the 2PC protocol, without using heuristics (failing during commit) which would be very bad!! So should work for all XA compliant RMs.

                   

                  Potential issues:

                   

                  1. Doesn't work if the resource we are testing is also one-phase aware as there can only be one in a transaction.
                  2. I'm not 100% sure that handling of one-phase resources is mandated in the JTA spec. This needs checking if you want this to work on other application servers.

                   

                  Paul.

                  • 6. Re: Arquillian transaction support
                    paul.robinson

                    Also,

                     

                    It looks like your transactional support assumes you are using XA Datasources. Is this correct?

                     

                    I suspect a user would get unexpected behavior if they used a non-XA datasource with these annotations. For example, the rollback would not occur as the resource is never enlisted in the transaction that eventually rollsback.

                     

                    This is a potential problem for your "it just works" requirement.

                     

                    Paul.

                    • 7. Re: Arquillian transaction support
                      dan.j.allen

                      "What are we testing here?" is a great place to start. After all, there's no use spending time adding features just to boast that we have them

                       

                      The request for this feature can likely be traced back to the testing chapter of the Spring reference docs where it introduces transaction management. I think this section is misguiding the developer by making some pretty bad assumptions about what can be legitimately asserted.

                       

                      The first sentence is the source of the widespread fixation that "an integration test should never change the database".

                       

                      One common issue in tests that access a real database is their affect on the state of the persistence store.

                       

                      That should not be an absolute. There are cases where that should be true, and cases where it shouldn't. But here's the real misinformed statement.

                       

                      Also, many operations — such as inserting or modifying persistent data — cannot be performed (or verified) outside a transaction.

                       

                      What? Actually, I'd say it's the opposite. We can't be sure inserts or updates happened correctly *unless* the transaction completes. It's likely better to have a compensating transaction than to naively rollback with a single participant.

                       

                      The next paragraph goes on to suggest that assertions within the transaction will all be accurate and you can just rollback when you are done. However, we know that with ORM this is certainly not the case since it queues DML (insert/update/delete) statements until a flush (usually at commit time unless something triggers an interim flush). Assertions in this context could be nothing more than false positives.

                       

                      On the other hand, if the goal is to detect mapping errors without affecting the state of the database, then we need to have something along the lines of your idea of a mock participant voting to rollback the transaction. We'll call it the "dry run" use case. You want the persistence provider to attempt the SQL statements to be sure no prerequisite (or the query itself) fails.

                       

                      Here's the statement that is at most odds with the Arquilian mission (and most appaling to us):

                       

                      If you want a transaction to commit — unusual, but occasionally useful when you want a particular test to populate or modify the database — the TestContext framework can be instructed to cause the transaction to commit instead of roll back via the @TransactionConfiguration and @Rollback annotations.

                       

                      Unusual? Certainly not. That's called a real test And it's what we prefer, actually. It's better to give the developer the opportunity to clean up the database either by properly rolling back the operation or by providing an easy way of doing a compensating operation.

                       

                      To move forward, I think we should define the different use cases and create some hypothetical scenarios to match. Here's my first cut at identifying them:

                       

                      1. Basic - Wrap the test in the context of a transaction so that transaction components can be used safely, committing afterwards. If the state of the persistence store changes, so be it.

                      2. Dry-run - Force the SQL statements to be prepared and executed to check for mapping errors, validation errors or invalid SQL, but don't alter the state of the database

                      3. Compensating - Allow the test to modify the state of the database, but provide a callback (or script) to revert the changes after the test assertions are checked

                       

                      Can you think of cases I missed? Should we break these down furthur.

                       

                      Part of our job will be to undo the misconceptions planted in the minds of developers by the Spring reference documentation (btw, even as a Spring developer I saw the holes in those recommendations).

                      • 8. Re: Arquillian transaction support
                        jhalliday

                        hmm, interesting discussion.  I can see the utility of an annotation that allows a test method to run in a transaction. I can even see the benefit of allowing commit/rollback of that tx to be configurable through the annotation. I think you've jumped from that general framework to a deep discussion of the issues in supporting the specific case or ORMs, which are just one type of transactional resource manager, albeit a common one. So, there certainly needs to be some consideration of how e.g. message queues or distributed caches may behave.  However, I'll stick with the JPA discussion thread for now.

                         

                        The separation between SQL execution and transaction commitment is certainly key. Spring incorrectly lumps these in together, with obvious drawbacks. Paul has identified various interesting hacks, but for a transaction manager neutral framework like arq none of them are particularly attractive. I think the best compromise for implementing 'dry run' is probably to have the test framework explicitly flush the JPA session, which should be sufficient to force the queued SQL to be executed. That's normally all a beforeCompletion will do, but there is nothing to stop the call being made directly instead.  I also like the 'compensating sql' hook. Sure using a @before/@after to reset the entire db is an option, but it's expensive. Being able to apply just a test specific minimal change set to the db would help execution time.

                         

                        Whilst an explicit flush will work for JPAs, a more general solution is needed for other types of RM. At present there is no resource ordering guarantee in the JTA spec, so to make Paul's suggestion work you'll wind up needing a plugin for each transaction manager you want to support.  Maybe start out with the generic JPA support and add the more complex general XAResource support later if there is demand.  Looking ahead though I think it makes sense for the transaction annotations to be separate from JPA/database package, they are not specific to that one class of resource manager.