I've seen a deadlock several times when booting a JBoss server under load. I'm still not sure if this is a JBoss problem or problem with the IBM JVM. This is JBoss 4.3.0 on AIX 5.3 running Java 6. Full JVM version:
java version "1.6.0"
Java(TM) SE Runtime Environment (build pap3260sr8-20100409_01(SR8))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc-32 jvmap3260sr8-20100401_55940 (JIT enabled, AOT enabled)
J9VM - 20100401_055940
JIT - r9_20100401_15339
GC - 20100308_AA)
JCL - 20100408_01
I have taken several thread dumps when this occurs and attached the analysis of one of them. The others showed the same symptoms. I've scrubbed out our application's class and package names and replaced them with our/pkg/*.
At a high level, the app should complete almost all of its useful work with the ajp-0.0.0.0-57001-* threads. Instead, all traffic to the 57001 port hangs. The thread dump shows that these threads are waiting for Oracle JDBC classes to be loaded.
It appears that the thread which is trying to define the Oracle class that the other threads are waiting for does not make any progress and is waiting for notification on the internal VM class table lock, which is unowned. That notification appears to never come or is lost.
Has anyone seen this kind of behavior before? I have not been able to find it in the JBoss forums or the JBoss issue tracker, nor with Google.
The attached analysis has more details and some of my conclusions.
Cheers,
Patrick
