beforeCompletion is called successfully just before this WARN message. However afterCompletion is never invoked and instead the above message pops up.
I am not sure if this is linked with - https://jira.jboss.org/jira/browse/JBTM-425
No, these are not linked. The JIRA details a case where afterCompletion processiing is not attempted. In your case afterCompletion processing is attempted but fails.
You will need to obtain more information here. The message with tag xxx.TwoPhaseCoordinator_4 is only printed when the afterCompletion synchronization processing calls a registered synchronization and it throws an exception. I suggest you use a debugger to intercept the afterCompletion processing call and see where the exception is coming from. Break method afterCompletion in class om.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator and see what happens. You can find the right place by searching for the xxx.TwoPhaseCoordinator_4 tag.
There are known issues with the EJB container expecting the Synchronizations to be invoked in a certain order, which is a mistake as the transaction manager makes no such guarantee. If the problem is not fixed in AS 5.1.0.GA you'll probably need to talk to the EJB guys.
Just tried that, installed JBOSS 5.1.0GA and that does not seem to have fixed the problem. In fact with JBOSS 5.1.0 GA the below warning message that afterCompletion failed itself has disappeared, and it has not been called either.
I will try to run it through a debugger and see if I can find some more information.
Let's also be clear: if this is related to applications assuming a specific ordering for synchronizations (or 2PC participants) then those applications are broken. The standards do not define an ordering of any participant within a transaction. Of course we have extensions to allow ordering to be controlled, but those are outside the standards and hence not portable. If ordering is required then it should be dealt with within the participant, e.g., by enlisting a proxy participant that internally calls the real participants in some pre-determined order. It's easier to do for Synchronizations where there's no recovery implication, but not something I'd recommend for 2PC participants.