If you use an MDB, if the message fails it should go to the DLQ, you can configure this to happen after one try. (This isn't transactionally safe, from what I've heard.)
Alternatively, you could check the 'redelivered' flag on the message in an MDB. So:
1. MDB receives message
2. Do database processing, fails, roll back
3. Message redelivered ... instead of retrying the DB transaction, add the message to a different queue, consume the current message.