JBoss 6 100% CPU usage after scanning
nick.x.newman Sep 22, 2010 1:04 PMHi,
My instance of JBoss is subjected to external scanning to check for vulnerabilities, and after these scans it often ends up in a state in which it seems to work fine, but uses just over 100% of a CPU (on a 4 CPU box) thereafter. I'd like to avoid this unnecessary CPU load.
The first thing I did to investigate was to use the JMX console to look at the thread CPU usage. That showed that threads named pool-2-thread-2 through pool-2-thread-13 had very high CPU time (e.g. 2917560 ms and rising), so it's clear that these are the problems. (But pool-2-thread-1 had only 1490 ms used, for some reason.)
Oddly, when I ask the same MBean to display a thread dump, these threads seem not to be there. So I obtained a thread dump using kill -3 uid (it's on a Linux box) . To avoid cluttering things up I have included that dump as an attachment rather than pasting it all here. However, here is a sample of one of these threads. It seems to be typical of this pool-2-thread-x group except that thread-4 is "runnable" rather than "waiting on condition".
"pool-2-thread-13" prio=10 tid=0x082e4000 nid=0x58dc waiting on condition [0xbed9e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xd8a9ce60> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2061)
at org.jboss.threads.QueueExecutor.takeTask(QueueExecutor.java:725)
at org.jboss.threads.QueueExecutor.access$400(QueueExecutor.java:45)
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:814)
at java.lang.Thread.run(Thread.java:619)
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
But having got this far I'm not sure how to proceed. If anyone has any ideas (other than telling my company to quit being so paranoid and to lay off the scanning) I'd love to hear them.
Thanks,
Nick
-
ThreadDump.txt.zip 4.9 KB