As a member of the JBoss Transactions team, I've spent some time looking at the transaction extension. From what I have seen so far, I can certainly say that it looks interesting! I have a few queries that I'd like to raise here and also propose some areas where we could contribute to the project.
1. What is the purpose of rollback after the test completes and do you have a specific use-case? I think some people use it to undo any work done by the test, leaving the DB clean ready for the next test. The problem with this approach is that it doesn't provide a real test as it doesn't test the ORM->DB interaction. DB specific things could be done to ensure the DB is interacted with (trigger a flush), but this is specific to the scenario where the RM is a DB. Also, this flush behaviour is probably better placed in the Arquillian Persistence Extension, rather than ATE. We discussed this in more detail, almost a year ago here, and didn't really come up with a satisfactory conclusion.
2. How do you revert the test resources after a test, if you instruct it to commit? It can't be done in the @After or @Before as they run in the same TX. If you update a resource in the test and then negate the update in @After, you would leave nothing to commit when the transaction commits. Another problem with this, is that you may need to consume a message, produced by the test, in order to tidy up from the test. This message will not be available until the after the transaction has committed, and thus not available in the @After.
Maybe we need some general support for compensation or setup that happens after or before (respectively) the test in its own transaction? I see you have this support in the persistence extension by way of specifying a yaml file that states what the DB should look like before the test runs (Which, BTW, I assume runs outside of the test's transaction?). This is great for databases, but not for other types of RM.
3. It doesn't look like you can combine different transaction types in the same test suite, due to the fact that only one type can be present on the classpath at once. This would be a problem for our TXBridge tests that bridge a JTA transaction to a WS-AT transaction and visa-versa. Here some tests start a JTA transaction and others start a WSAT transaction. Of course, we would need a WS-AT plugin for ATE for this to be an issue: more about that later.
4. I don't think you can assert anything after the transaction has completed. Clearly you can't assert anything in the test method for checking the state after the commit/rollback, but also you can't assert in the @After method (which is probably bad practice anyway?) as it is executed in the same transaction as the test. Maybe we could add an attribute to the @After and @Before to state that it should be executed in either the SAME, NEW, or NONE transaction?
However, I do wonder whether application developers need to assert anything after the test's transaction has committed? As a transactions middleware developer, I would need that facility because I am testing the Transaction Service. Application developers can assume the transaction service works, so don't need to assert after commit/rollback? WDYT?
5. A test can only run 1 transaction. We have some tests that run two transactions. Again this could possibly only be an issue to transaction middleware developers, rather than the vast majority of the target audience.
6. When implementing a transaction extension you have an @TransactionScoped injection which seems to be used to squirrel away the UserTransaction and have the right one made available during the lifecycle of the test. Can someone explain how the lifecycle of the @TransactionScoped variable is managed by ATE? Is it tied to the transaction lifecycle? If so, I can't see how this is managed as ATE doesn't know anything of the actual transaction type (JTA/Spring/WS-AT etc). Maybe I have the wrong end of the stick here?
As I mentioned earlier I'd be happy to contribute some developer time to this project. In particular we were thinking that an XTS transaction type (WS-AT and WS-BA) implementation could be useful for our users. I assume this impl would live in the JBossTS project?
Also, we noticed that you don't yet have a quickstart for the project. I can contribute one of these if you'd like?
I'll also continue to review this project and come back with any feedback/new-features etc.