jboss cpu over 100% usage

[jblbecarelli]

Hello

we have an application with jboss 5.1.0GA .

Sometimes inexplicably the jboss  process use over 100% cpu processor.

We must stop jboss and restart.

I have attached some thread dump ,

threads_ok.html is when all is ok .

theads.html - threads2.html - threads3.html are thread dumps when cpu processor is over 100% usage .

I had notice that some process are blocked in threads2.html but i don't understand if for db problem or application problem.

Who can help me ?

Best regards

[vaedama]

Hello Luca,

Please enable JBoss level logging. Set the Logging level preferably to DEBUG and post your stack trace.

I can assume a wide range of reasons for this problem.

I can provide you some solutions:

1. Keep all your code in try block and destroy all the datasources you used in each thread in the end of finally block. (You can use C3P0 jar). Alternatively, set the datasources to null in the end of finally block.

2. Increase your thread count in your application configuration. (You can set this mostly in properties file/config file of your application, not JBoss properties).

3. Kill the threads soon after their usage is done. (Just for testing/debugging purpose)

If you have any daemon threads be careful while you are dealing with them.

-Sudheer

[jblbecarelli]

thanks very much for your answer

solution 1 is for me problematic because i use seam and jta managed from container

i have enabled in jboss-log4j.xml 

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="true">

when the problem happens i rebuild thread dump and log ,

at this moment , when jboss go over 100% cpu , i have in log many

org.jboss.seam.ConcurrentRequestTimeoutException: Concurrent call to conversation

but i presume that not is the primary problem , this happen because server go slowly

you presume that the problem is in db connection ?

[vaedama]

Clearly JBoss is not releasing its resources (threads). I am not speaking about the sql connections. But try to initialize all your datasources if possible because I ran into a similar issue long ago. You can check if the connections are being released or not from the JBoss JMX Console --> jboss.jca Verify if all the connections created are being released or not.

Did you try solutions 2 and 3? Kill the threads in the finally block of your run() method. That may help. Solution 2 (Increasing the thread count) may not solve the problem. But we may figure out the reason from this. Try to increase your thread count to 10 and see how much time it takes to reach 100% CPU utilization.

Then increase your thread count to 50 and see how much time it takes now. I will suggest not to over-populate threads unnecessarily. This is just for your testing.

Meanwhile, I will think of other possible solutions.

[jblbecarelli]

sorry , but how i can increase the thread count ?

[vaedama]

Hello Luca,

Check your properties file in your project configuration.

If you are deploying your application war/ear in JBoss, you will have this mostly under ProjectName\war\WEB-INF\lib\LucasProject.PROPERTIES File.

If you have a parameter for thread count in that PROPERTIES file increase it.

If you don't find it there, locate the place where the threads are getting spooled from.

Find how many threads are active before your application is launched.

Find how many threads are generated after you launch your project.

Find how many threads are being generated when your CPU utilization is 100%.

Re-run the test.

If the thread count is same in your tests when the CPU utilization is 100% then I think, something is wrong with your application configuration.

-Sudheer

[jblbecarelli]

Probably i have found the problem,

For check if an image exists , i create an HttpURLConnection and try-catch FileNotFoundException.

If i don't urlConn.disconnect(); the number of connection increase and after some hours one thread of jboss go over 100% cpu.

Thats all.

Sudheer, thanks for all your reply.

Luca