Help needed on JAVA Options settings for JBOSS4.2.3GA

[pradeepn]

=======================================

System spec:

MS Windows Server 2003 (SP-2)

Intel Xeon CPU @ 2.27GHz, 4GB RAM

JBOSS 4.2.3.GA

JAVA version 1.6.0_06

Web Application (Warfile)

========================================

JAVA Options current settings as below:

set JAVA_OPTS=%JAVA_OPTS% -Xms256m -Xmx1536m

Question: My application was freezing in Production  with above java options. For my System Spec. :

1> can I use this JAVA options as shown below ?

set JAVA_OPTS=%JAVA_OPTS% -server -Xms1024m -Xmx1024m  -XX:PermSize=256m -XX:MaxPermSize=512

-XX:+UseConcMarkSweepGC -XX:+ExplicitGCInvokesConcurrent

2> Or Based on my System Spec. what JAVA settings is prefered so that my application avoids(minimizes) freezing @ Production envi ?

Thank You.

[peterj]

Are you certain that your application is freezing because of garbage collection issues? How do you know? If that isn't the problem, then switching garbage collectors won't be the answer. Have you grabbed the garbage collection stats and analyzed them? Also, if you are truly running out of heap space such that major collections are taking up all of the processing time, then decreasing the heap max size from 1.5G to 1G will not help!!!

Also, you say the app is freezing. I assume you mean it is hung. Have you taken an Java thread dumps? Those will usually help you pinpoint areas of you code that are causing the hang (things like database deadlocks, synchronization deadlocks, or infinite loops).

[pradeepn]

I read in one of your POST that web application sometimes results in GC issues.

Yes now I'm trying to monitor the issue with JProfiler6. I'll install analyze and get back to you.

Can I temporarily fix the issue with above given settings by increasing heap max size from 1.5 to 2g ?

[peterj]

Maybe. maybe not. It all depends on the root cause of the issue.

[pradeepn]

I have posted both server log and GC log.

1> I analysed the server log there are few minor Errors neglecting that can you please tell me what major issues causing the server to freeze.

2> Also initialy GC logs I observed Full GC is happening at irregular interval times, is it due to Memory leak. Can you please let me know your observation on this.

Thank You.

[pradeepn]

I was able to collect one more GC log yesterday..... Can anyone help .

I'll take Java thread dumps shortly and post.

Thank You.

[pradeepn]

Here is JAVA Heap dump from JBOSS  using JMX Console

[peterj]

Garbage collection is not the problem - even the major collections are averaging 1/3 of a second with the highest being about 500ms. Heap size is not the problem - the heap in use after major collections is a small portion of the full heap.

However, it does look as if some code is calling System.gc() which is causing major garbage collections even when they are not needed. For example, near the end of both of the GC logs you posted the heap is less that 1/4 full when a major GC happens. You need to either find that code and remove the System.gc() call or add the -XX:+DisableExplicitGC option to the JAVA_OPTS in run.conf.bat.

I'll look at the thread dump when I have some free time.

By the way, you might want to look at my white papers on how to analyze GC logs:

Java Garbage Collection Statistical Analysis 101

Java Garbage Collection Performance Analysis 201

at http://www.cmg.org/cgi-bin/search.cgi?q=java+peter+johnson&x=30&y=10

[pradeepn]

Thanks Peter Johnson for your help in the recent past, U are always there as helping hand for all of JBOSS user. ALL hail Peter Johnson.

Since u were busy helping others i was able to take Thread dump from production envi. for two days (yesteday I posted 22April2013 JAVA Thread dump and other java Thread dump taken on 23April2013 posting now  )

1. I was able to notice there are few threads which are in "threadState:WAITING" is this causing problem?, i'm not able to find which are those . Is there any way I can find which of the features in my application is causing the problem. can I use Eclipse Memory Analyser tool ?

2.  Finding the code which is calling System.gc() would be tough I would rather add -XX:+DisableExplicitGC to run.bat, Thank you

I'm attaching another JAVA Thread dump taken on 23April

[peterj]

"ThreadState: Waiting" just means that the thread is not currently waiting on some event before it can be run. Most of those threads are threads in the thread pool that have not been assigned to requests. Ignore them.

What you need to look for it threads where the stack traces are fairly large. In the latest thread dump, notice that all of the waiting threrads have fairly small stack traces - 3 to 8 methods. Then notice this thread:

Thread: http-192.168.2.159-8080-2 : priority:5, demon:true, threadId:59

That thread appear to be the one generating the stack track. Notice that the classes in the stack trace are all from JBoss or Java.

What you are really looking for are threads where the stack traces include your code, or the code for deployed applications. As far as I can see, there is only one:

Thread: http-192.168.2.159-8080-8 : priority:5, demon:true, threadId:66,

You need to find out what is going on in that code that is causing thep roblem. Usually it helps to take 2 or 3 thread dump a few seconds apart - that should hopefully show you where the code is hanging (look for the methods or lines that have changed in the stack trace - if nothing changed, then in this case look at the database or suspect an infinite loop in your code)