we are having problems with our application running on jboss 4.2.3. I'm not sure if the transaction forum is the right place to ask this - i tried asking on the jboss as forum in an earlier stage of the problem but got no replies.
What happened was that we noticed that every once in a while, our application stopped responding to http requests. It turned out that all database connections were in use. We figured out that we didn't have a query timeout set. We set it, added a full thread dump on a transaction timeout and tried again.
The next time, our application crashed because every thread in the thread pool was busy waiting for the QueuedPessimisticEJBLock. I found https://jira.jboss.org/jira/browse/JBAS-6348 (QueuedPessimisticEJBLock checks for MARK_AS_ROLLBACK only, failing to detect a transaction that Arjuna already rolled back). I fixed it and deployed it. Now our application no longer crashes but detects a timeout and reports it - at least one problem fixed and a Jboss bug confirmed.
However, we're not there yet. What we have now is a situation where a bean is locked indefinately. Every process that tries to access that bean tries to wait for the lock and gets a transaction timeout. Because we have a few web service clients which will retry a small operation every minute if it failed which locks this specific entity bean, we see timeouts every minute. The only way to fix the problem is a restart of our application/jboss.
Because we have a transaction timeout of 5 minutes and a query timeout equal to the remaining time in the transaction, i think a lock should never be held for more than 5 minutes. Is there a known issue where this behaviour happens? Is there a way to monitor locks in a running (production) environment, for example with the jmx-console?
We did find that QueuedEJBPessimisticLock has a possibility of calling
with no check on txTimeout being 0. We're not sure if this is ever set to 0, but that would mean an indefinately long wait, unless notify() is called by another thread. Could this be the source of our problem?
Our configuration is: EJB 2 with CMP, commit option A and QueuedPessimisticEJBLock. This entity bean has all its read-only methods set as read-only.