The jboss.mq implementation of XAResource.recover behaves in a non-standard, and IMO invalid, way, in that it ignores the value of the flag argument.

The XA specification explains why the flag argument is there but it isn't necessarily needed in Java since we can we can return ALL the recoverable XIDs in one fell swoop.

However, code that uses the flag "correctly" does not work with jboss.mq - this is how the XA spec says you should call recover:
1 x with TMSTARTSCAN
If fewer than the max are returned, you're done, otherwise:
n x with TMNOFLAGS until fewer than the max are returned
1 x with TMENDSCAN

Note that in the C API of recover the caller passes in the value for max, but there is no such argument in the Java API. It appears to be undefined if there is a maximum but clearly there doesn't have to in Java, and yet we have the flags argument.

Since there is no max argument in the Java version of recover, other Java implementations (eg. Websphere MQ) of recover have taken the view that they will return ALL the XIDs when called with TMSTARTSCAN and return none with any other value. This means you can use the above logic without caring what max is or even if there is one.

But, if you use this logic with jboss.mq you enter an infinite loop because it returns all XIDs for both TMSTARTSCAN and TMNOFLAGS. This means that for jboss.mq you MUST know to only call recover once. This is unnecessarily restrictive and not general, and makes it hard to write a general external transaction manager.