9 Replies Latest reply on Sep 2, 2010 9:48 AM by tsmets

    Trying to increase performance, doesn't seem to be CPU or DB

    jamicide

      Hello

      I am trying to maximize the performance/thoroughput of my JBossAS.

      Currently my server processes approximately 60 external requests per second. At this rate CPU usage is at ~50% usage, memory is at ~35% and the load on the DB machine is similar.

      I have tried fiddling with various parameters of the system, namely the JMS threadpool size (all requests are placed on the JMS queue) and the JMS connection pool size. The modifications I make only seem to have a minor effect on the system's performance, and even setting the pool settings to very high values doesn't result in a much different operation. The system loads never goes higher than what I mentioned above.

      Are there settings which I could change in addition (or in place of) those I mentioned to get the desired results? Is there potentially another bottleneck (local secondary storage, network, etc.) that could cause this?

      Any help is greatly appreciated.

      Server specs:

      jboss-4.2.2.GA
      Dual-Quad Core System, 4G ram

        • 1. Re: Trying to increase performance, doesn't seem to be CPU o
          peterj

          Try tuning the JVM garbage collector. See my presentation at http://www.cecmg.de/doc/tagung_2007/agenda07/24-mai/2b3-peter-johnson/index.html.

          I also have another resource on JBossAS tuning, but it is not free. If you like I can provide the URL.

          • 2. Re: Trying to increase performance, doesn't seem to be CPU o
            jamicide

            Hi Peter, thanks for the response.

            I ran several samples using the method you outlined, summing the GC times from that one column.

            Typically the GC took about 2% of the total running time of the test. For example, my latest test ran for 75 seconds, and the garbage collector ran for ~1.5 seconds of that time.

            The memory settings I am using are:

            -Xms3072m -Xmx3072m -XX:NewSize=1024m -XX:MaxNewSize=1024m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000

            This seemed like fairly good results to me - what do you think?

            • 3. Re: Trying to increase performance, doesn't seem to be CPU o
              tsmets

              Peter,

              You seem to use custom made tool to analyse the GC. At least it is what I can gather from the graphs in the slides and from the fact that you indicate the regular expressions to parse the GC vs FullGC.

              So my question is what is :
              Do you use custom made tools or tools like jvmstat ?

              Can you explain what is yr approach ?

              \T,

              • 4. Re: Trying to increase performance, doesn't seem to be CPU o
                peterj

                jamicide, are you sure you need 3GB of heap? Graphing the results should help you determine that. No sense using more memory that what you need.

                tsmets, I wrote a simple Java app that uses a regular expression to gather the verbose:gc data and convert it to a CSV file. I then read the CSV file into a spreadsheet app (Excel or OpenOffice.org Calc) and use its graphing features to chart the results.

                Here is my Java app:

                import java.io.*;
                import java.util.regex.*;
                public class Analyzer {
                 public static void main(String[] args) throws Exception {
                 InputStream fin = new FileInputStream(args[0]);
                 int iSize = fin.available();
                 byte mvIn[] = new byte[iSize];
                 fin.read(mvIn, 0, iSize);
                 fin.close();
                 String strText = new String(mvIn);
                 PrintStream fout = new PrintStream
                 (new FileOutputStream(args[0] + ".csv"));
                 fout.println("Before,After,Seconds");
                 Pattern p = Pattern.compile
                 ("\\[(?:Full |)GC (\\d*)K->(\\d*)K\\(\\d*K\\), ([\\d.]*) secs\\]");
                 Matcher m = p.matcher(strText);
                 while (m.find())
                 fout.println(m.group(1) + "," + m.group(2) + "," + m.group(3));
                 fout.close();
                 }
                }


                • 5. Re: Trying to increase performance, doesn't seem to be CPU o
                  jamicide

                  Hi Peter

                  no you are right. I loaded my output into GCViewer (http://www.tagtraum.com/gcviewer.html) and the graph showed I didn't use nearly that much memory.

                  Will try lowering this.

                  • 6. Re: Trying to increase performance, doesn't seem to be CPU o
                    jamicide

                    I lowered the memory and am getting the same performance as before (as expected I suppose).

                    Any other ideas as to what is limiting the system?

                    Thanks

                    • 7. Re: Trying to increase performance, doesn't seem to be CPU o
                      peterj

                      Performance tuning is extremely over a forum. If you are not comfortable with, or know how to go about, performance tuning, you need to hire someone to come and tune your system. There are just too many possible things to look at.

                      • 8. Re: Trying to increase performance, doesn't seem to be CPU o
                        kjkoster

                        Dear Jamicide,

                        I spent a little time with the book "System Performance Tuning" by Mike Loukides. While is speaks of VAX machines and early Sun boxes, the lessons in that book are universal and I applied them to Windows XP with success. :)

                        Once you get beyond the reasonably obvious hogs, tuning gets tricky fairly quickly. You go into the lands of memory bandwidth, inter-cpu communication and lock contention.

                        You need to find out what subsystem is bottlenecking: cpu, memory or I/O in its various forms. Those are things we cannot help you with, since we have no access to your machines.

                        Interestingly, the list of resources in a machine is rather short; cpu, RAM, network I/O and disk I/O (for each disk). Just check each of these in turn. If all of these are used well below their capacity, you are likely to have synchronisation issues in your Java application.

                        Kees Jan

                        • 9. Re: Trying to increase performance, doesn't seem to be CPU or DB
                          tsmets

                          It is sometimes better to have 2 JVM running in parallel instead of one...

                          You should also consider the type of HW architecture you have. Multi CPU, multi cores, ... if you look at the GC algorithmes available since 1.4.1 (talking back to 2003... you might also choose a better one ).

                          Apparently Sun (Oups ... excuse me Oracle) provide a fairly good article http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html

                          One thing I learned from the people maintaining the systems alive ...

                          There are very few people that can really help you with a theorical answer.

                          Just try all the settings that look consistent to you and watch them behave, live !