GARBAGE COLLECTOR PROBLEM

[mlelola]

Hi, I have a problem with the gc performance in production.

The enviroment is the next:
Linux Red Hat 9.0
biprocessors
2 Gigas of memory
Jboss 3.2.3

I am using these parameters:
-Xmx1536m
-Xms512m
-Xss1024k
-XX:NewSize=96m
-XX:MaxNewSize=256m (igual que NewSize)
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:ParallelGCThreads=2


About 19:00-20:00 pm are the hours with more users but depend of the day at 22:00 pm or 1:00 am or 18:30 pm... It doesn´t happen at the same hour and It doesn´t happen daily, the system begins to answer slowly and finally we have to restart jboss.

Looking de gc log with the tool "hpjtune", we can see that there are a lot of calls of gc (scavenge and full gc) and there are an increasing heap size with each gc.
The system calls to old gc and after a a old gc, the head size drops to a much lower level, but the level of heap size is getting bigger and finally all gc are only old gc. There are not any scavenge gc.
Finally we have to restart jboss when the heap size is about 700M.

If there is not any problem, the graphic shows a scavenge gc and full gc (50% each one), not old gc, and the maximum level of heap size is 200M, and when the users are not using the system, the level of heap size is 50M.

Could anybody help me?

Thanks in advance[/img]

[thoennes]

In the first quick browse, I noticed that parallel GC does not make much sense with just 2 processors. It may even slow down things a bit. Use it if you have 4 procs or more.

But this is not the cause of your problem.

In addition, the per thread stack size (-Xss:1024k) seems quite big. How many JVM threads are running?

Did you watch the memory usage of the JVM (VMSIZE, RSS) if it gets slow? Do you see increased paging activity? The -Xmx size should be small enough to fit in memory all the time. If the heap has to be swapped out, things get much slower.

What are your users doing?

In summary, the information you provided is not enough to do any useful analysis. Basically, its not a specifc JBoss problem, but a general JVM tuning one. Unless you found a memory leak; in this case you should first upgrade to a recent JBoss 3.2.x version.

Cheers, Jörg

[mlelola]

We have been playing around with the parameters that I explained before.

We changed Xms to 1024m and after several hours running with these setting, we obtained this:
java.lang.OutOfMemoryError: unable to create new native thread

Then we decided to reduce Xmx to 1024m instead of 1536 and Xms to 880m.
We have 2GB of memory and I think that the act of reducing the maximun Java heap size will
made the "native heap" bigger and this change should prevent any OutOfMemoryErrors based on the
native heap.

Our server only has Jboss. The 2GB are for Operative System and Jboss.

Now, we are testing these setting;
-Xmx1024m
-Xms880m
-Xss1024k
-XX:NewSize=220m
-XX:MaxNewSize=220m
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:ParallelGCThreads=2
-XX:SurvivorRatio=8.

Could anybody tell me if these settings are ok????

We had been testing our application with JProve an JProfiler and we hadn´t found a memory leak.

About Xss exactly I don´t know if 1024k is too much or not. We are using the values of ulimit by default.
In our application we are creating threads to do specific functionality,
but I don´t know if we are using a lot or not.

We have 3 problems:

1. OutOfMemory: unable to create new native thread: I have explained it before.
2. StackOverFlowError: This appears sometimes and the Jboss dies.
3. Anormal behaviour of GC. I have explain it in the previous topic.

[jiwils]

OutOfMemory: unable to create new native thread.

This error is actually not necessarily related to lack of memory. It basically means that the JVM could not create a new thread of execution as requested.

I have only ever seen it in relation to the Linux kernel 2.4's (and earlier) use of lightweight processes instead of true "threads". 2.4 has a hard-coded limit of 1024 processes that a process can spawn/create. So, because a thread is a process in 2.4 (and earlier), you can only create 1024 in your Java program. This limitation is not just a Java limitation however. I have read that you can recompile the kernel to increase this hardcoded limitation, but I have not done that. For Kernel 2.6+, the use of the NPTL (Native Posix Thread Library) which allows for more "true" thread behavior, will allow you to overcome this limitation (so I have read).

You can monitor the number of threads used by your process by using the JMX console in conjunction with the ServerInfo MBean's listThreadDump method. See if you start seeing this exception if your thread count gets up above 1000. If so, then this kernel limitation is the likely source of this particluar exception.

In applications where I had this problem, I limited various thread pools so that I never reached the process-wide 1024 thread limit, and that seemed to work well.

I am not sure how to address you other problems, but I hope this information helps.

[thoennes]

This lets me think of the following scenario:

If you actually reach the thread limit of 1024, which the error message "OutOfMemory: unable to create new native thread" indicates, the your threads consume 1024 * 1024k of memory for the per process stack.

I.e. in addition to the heap of 1024MByte your are using another 1024MByte for the stacks. So all your real memory is filled by heap and stack of the JBoss JVM.

Furthermore, the text segments and the other processes also need some space on the machine, and so the machine will definitely start swapping, slowing down things a lot.

On the other hand, a StackOverflowError may indicate that the stack is too low. But I think you should avoid this by other means.

In summary:

1. Try to increase -Xss to the default value (64k?).

2. Monitor the number of threads and try to reduce thread usage.

Cheers, Jörg

[shravanxm]

I think there is no use of using -XX:+UseConcMarkSweepGC with -XX:+UseParNewGC.

The first option for GC will clean only the tenured generation with low pauses on the app. The second option will also do the same but it also cleans the eden space using parallel threads.

The parallel threads parameter is set to 2. Can I know how many CPU's does the machine have?.

Setting -Xss to 1024 is very high. Also no need to set it if the value you want is 1024, since default is 1024 for stack sixe in linux 64 bit OS.

Try using 256k. This is ideal in most cases.

To get better performance, don't force JVM to resize the heap size. If you know how much memory your application needs, set the same for min and max sizes. If the application(s) will load the classes, use -XX:PermSize and -XX:MaxPermSize.

Also, try using -XX:NewRatio=3 instead of new size and max new size. This will allocate 1/4th of the memory for eden space.

Correct me if any of my suggestions are wrong.

Thanks    

[peterj]

shravanxm, why are you responding to a 6-year old thread?