A question about JVM activ thread count

[thomas2008ch]

I installed the applications on Jboss Application Server which installed on LINUX. I noticed that every time if the so-called jvm-activ-thread-count goes high, the Jboss server will crash.

Someone knows why? how can I prevent the jvm-activ-thread-count going high?

[peterj]

Where are you finding the "jvm-activ-thread-count"?

[thomas2008ch]

I use a monitoring tool which can show the  jvm-activ-thread-count. You can also programm this in your java code.

[peterj]

Without knowing how that tool calculates the active thread count, and without knowing what is going on in the application server at that time, I am afraid I cannot help you.

You might try taking a thread dump (or several spaced a few seconds apart) and look at those dumps to see if they tell you anything. Identifying the "active" threads is always the key - perhaps you have some errant piece of code the spawns threads and that code gets into a loop?

[vb_rich]

I have a similar issue, but more details to provide. I’m hoping someone may have an idea to help me triage. Looking for any/all ideas on how to research this.

Environment:

• AIX ??
• JBoss 3.2.7 (yes, very old)
• JRE 1.3.1 (yes, very old)
• J2EE application (Swing Applet, JSP, EJB, Oracle)

Thread Monitoring Solutions:

• Hyperic agent
• Twiddle (JBoss command line tool to access JMX)

Historic:

• For the past 7 years our application has hovered around 150-200 active threads.
• No new code has been released recently

Issue That Started Last Month:

• Operations person “accidently restarted jboss” in the middle of the day. They said they misread an SOP and made a mistake. No other information can be found regarding what else may have happened. I believe something was deleted/configured/moved/etc from past experience with our support group.
• Per monitoring, thread counts jumped up dramatically the moment JBoss was restarted and we have had a recurring problem since. We are forced to restart our system every 4-5 days.
• Since the incident, as soon as we restart jboss, threads start rising at about 17k threads a day (never going downward).
• It has reached up to 90,000 threads
• Our JVM begins to produce heap dumps as thread counts rise past 60,000
• The heap dumps become so large/frequent that is causes our system to crash

Research:

• Heap dump analyzers show no memory/thread issues in the JVM, but our java version is so old that we are not sure how accurate the analyzer is.
• Per Hyperic/Twiddle, thread counts seem to rise in coorelation with applications coming into the system. When we put up a maintenance page, thread counts stop rising (do not go downward either)
• Twiddle allows us to invoke “listThreadDump” on the “jboss.system:type=ServerInfo” mbean, which shows the thread count of all thread groups, but it does not break the individual thread groups down to see how many threads are in each group (so we can’t narrow it down)
• JBoss3.2.7 is so old that listThreadDump (on ServerInfo) is the only function we can execute for analysis at the JBoss level.
• Our QA system has received a heap dump, but all load testing we perform does not raise the thread counts past 150 in that environment. Also, the heap dump can’t be recreated consistently. Our system is “infinitely configurable” so we may not be hitting the correct scenario as many production clients may be.

Next Steps:

• Our next step is to failover to a secondary application server to rule out a hardware failure on the prod app server. We doubt this will help, but it will eliminate a variable.

Any suggestions for triage are welcome (other than the obvious advice of upgrading our versions). Ideas anyone????

[peterj]

>>Per Hyperic/Twiddle, thread counts seem to rise in coorelation with applications coming into the system.

By "appplications coming into the system" do you mean when you deploy additional applications? Or do you mean when additonal requests come in for already deployed applications? Or something else?

If you mean additonal requests, check the max number of threads in the server.xml file. (I think 3.2.7 has that file...)

Does the JVM let you take a thread dump (like you can with the Sun JVM) that lists each thread and the call stack for each thread? If so, take several thread dump as the thread count is increasing and determine which threads are new and what their purpose is. It is usually easy to distinguish threads that, for example, handle http requests from one that handle remote EJB calls.

Other than that, you might have to see what support you can get out of IBM to help your track down what is going on in the JVM.

By the way, you really should have started a new post since your question is not related to that of the otiginal poster.