Try tuning the JVM garbage collector. See my presentation at http://www.cecmg.de/doc/tagung_2007/agenda07/24-mai/2b3-peter-johnson/index.html.

I also have another resource on JBossAS tuning, but it is not free. If you like I can provide the URL.

Peter,

You seem to use custom made tool to analyse the GC. At least it is what I can gather from the graphs in the slides and from the fact that you indicate the regular expressions to parse the GC vs FullGC.

So my question is what is :
Do you use custom made tools or tools like jvmstat ?

Can you explain what is yr approach ?

\T,

jamicide, are you sure you need 3GB of heap? Graphing the results should help you determine that. No sense using more memory that what you need.

tsmets, I wrote a simple Java app that uses a regular expression to gather the verbose:gc data and convert it to a CSV file. I then read the CSV file into a spreadsheet app (Excel or OpenOffice.org Calc) and use its graphing features to chart the results.

Here is my Java app:

import java.io.*;
import java.util.regex.*;
public class Analyzer {
 public static void main(String[] args) throws Exception {
 InputStream fin = new FileInputStream(args[0]);
 int iSize = fin.available();
 byte mvIn[] = new byte[iSize];
 fin.read(mvIn, 0, iSize);
 fin.close();
 String strText = new String(mvIn);
 PrintStream fout = new PrintStream
 (new FileOutputStream(args[0] + ".csv"));
 fout.println("Before,After,Seconds");
 Pattern p = Pattern.compile
 ("\\[(?:Full |)GC (\\d*)K->(\\d*)K\\(\\d*K\\), ([\\d.]*) secs\\]");
 Matcher m = p.matcher(strText);
 while (m.find())
 fout.println(m.group(1) + "," + m.group(2) + "," + m.group(3));
 fout.close();
 }
}

Performance tuning is extremely over a forum. If you are not comfortable with, or know how to go about, performance tuning, you need to hire someone to come and tune your system. There are just too many possible things to look at.

Dear Jamicide,

I spent a little time with the book "System Performance Tuning" by Mike Loukides. While is speaks of VAX machines and early Sun boxes, the lessons in that book are universal and I applied them to Windows XP with success. :)

Once you get beyond the reasonably obvious hogs, tuning gets tricky fairly quickly. You go into the lands of memory bandwidth, inter-cpu communication and lock contention.

You need to find out what subsystem is bottlenecking: cpu, memory or I/O in its various forms. Those are things we cannot help you with, since we have no access to your machines.

Interestingly, the list of resources in a machine is rather short; cpu, RAM, network I/O and disk I/O (for each disk). Just check each of these in turn. If all of these are used well below their capacity, you are likely to have synchronisation issues in your Java application.

Kees Jan

It is sometimes better to have 2 JVM running in parallel instead of one...

You should also consider the type of HW architecture you have. Multi CPU, multi cores, ... if you look at the GC algorithmes available since 1.4.1 (talking back to 2003... you might also choose a better one ).

Apparently Sun (Oups ... excuse me Oracle) provide a fairly good article http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html

One thing I learned from the people maintaining the systems alive ...

There are very few people that can really help you with a theorical answer.

Just try all the settings that look consistent to you and watch them behave, live !