BasicAction.removeChildThread hangs

[normann]

We've just had a problem which I don't know what to do about. The situation was this:

A thread invoking a session bean which executed a stored procedure through JDBC seemed to hang. I suspect that it was the stored procedure that would not return.

Trying to fix the problem, we killed the database process so that the JDBC driver would receive an error and JTA could roll back the running 2PC-transaction. This didn't happen, though. Instead, our thread that has invoked the JDBC call blocked and we had to restart JBoss to get rid of it.

This was the (partial) stacktrace of the blocked thread:

Thread: Timer-10 : priority:5, demon:false, threadId:69, threadState:BLOCKED, lockName:com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction@34d50536

com.arjuna.ats.arjuna.coordinator.BasicAction.removeChildThread(BasicAction.java:909)
com.arjuna.ats.internal.arjuna.thread.ThreadActionData.purgeActions(ThreadActionData.java:254)
com.arjuna.ats.internal.arjuna.thread.ThreadActionData.purgeActions(ThreadActionData.java:230)
com.arjuna.ats.arjuna.AtomicAction.suspend(AtomicAction.java:351)
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionManagerImple.suspend(TransactionManagerImple.java:76)
com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.suspend(BaseTransactionManagerDelegate.java:155)
org.jboss.ejb.plugins.TxInterceptorCMT.endTransaction(TxInterceptorCMT.java:537)
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:361)
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
org.jboss.ejb.Container.invoke(Container.java:960)
org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)
org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)
$Proxy734.doSomething(Unknown Source)
...

Can anyone figure out what the problem could be (is it a bug?) or tell me if this is the expected behaviour?

Our setup is as following

- JBoss 4.2.3.GA
- PostgreSQL JDBC 8.2-509 driver.
- PostgreSQL 8.1 and 8.2 servers.
- Sun JDK 1.6.0u16 on Debian Linux amd64.

[jhalliday]

What's the thread dump from the thread that was holding the lock?

[normann]

If you're asking about a stacktrace of the thread that was executing the stored procedure before the database process was killed, this trackstace does not exist any more, it was lost in a browser refresh.

[jhalliday]

I'm asking for the stack trace of the thread that's holding the lock. That's not necessarily the one running the db query.

[normann]

As stated, the other stacktraces were lost in a browser refresh, so the stacktrace posted is the only one I have.

In the future if we see this again, how would I identify what thread is holding that particular lock? I suspect that only blocked threads show the name of the lock they're blocked on in the ServerInfo MBean's listThreadDump() output, and another thread holding the lock would not necessarily be blocked or probably blocked on another lock.

[jhalliday]

What's a web browser got to do with it? real developers kill -3 the jvm.

[normann]

At that particular point I was not logged into our server and therefore used ServerInfoMBean.listThreadDump in my web browser to get the thread dump instead.

If I had had access to the server, I would have used the jstack command which seems to do the same as your kill command. Now that I look at it again, I can see from the jstack output that it also lists the locked objects, so I'll take a look at the situation again if it arises.