Our transaction timeout is set to 1 hour.
1) Are there still deadlock situations that JBoss cannot predict or detect?
Yes. For example, when an instance is locked in a transaction then we call a method with RequiresNew and that new transaction tries to lock the instance that was locked by the previous not yet committed transaction.
A deadlock might also occur in other non-EJB layers like database and shared resources.
2) Is it possible, that even while using Instance Per Transaction with JBoss, for all Entity Beans to become locked?
Not in EJB layer except for the case above with when the same thread accesses same instances from different transactinal contexts (see above).
3) Is there a good way to debug this sort of problem?
You can increase log level to TRACE (see log4j.xml for example) for org.jboss.ejb.plugins and org.jboss.tm. And analyse thread dumps.