You are correct, this is a bug.
The code in SqlTableGatewayListener should be terminating the transaction after the poll.
Changing the isolation layer will allow for new rows to be inserted from another transaction but it will prevent modifications to those rows already read for the duration of the original transaction.
Can you please modify your version of SqlTableGatewayListener to release the transaction at the end of the poll and try again?
I will create a JIRA task for this.
I have created the following JIRA task to cover this issue.
I have updated code in SqlTableGatewayListener and SqlTableCourier.
It turns out that commit/rollback could have been called indirectly during the processing of a message so your suggestion should also have worked.
I have tidied up the code a bit so that all paths are now covered.