The 4.0.x JBoss manual, in its "EJBs on JBoss/Entity Bean Locking and Deadlock Detection/Troubleshooting/Hangs and Transaction Timeouts" section, states the following:

One long outstanding bug of JBoss is that on a transaction timeout, that transaction is only marked for a rollback and not actually rolled back. This responsibility is delegated to the invocation thread. This can cause major problems if the invocation thread hangs indefinitely since things like entity bean locks will never be released. The solution to this problem is not a good one. You really just need to avoid doing stuff within a transaction that could hang indefinitely. One common mistake is making connections across the internet or running a web-crawler within a transaction.

It's worth noting that JBoss Transactions, the new default transaction manager from AS 4.2.x onwards, does not have this problem any more as it rollbacks transactions when they time out.

This transaction manager can also be installed in AS 4.0.3SP1+, removing the previous JBoss transaction manager implementation.

Please visit the JBoss Transactions wiki and the JBoss Transactions home page in JBoss.org for more information.