7 Replies Latest reply on Nov 30, 2006 3:07 PM by jshrinivas

    Identifying performance bottlenecks

    jshrinivas

      Hello,

      I have a benchmarking application running on JBoss. The application gets requests from a client driver which keeps on sending requests at around 2000 requests per second. Although JBoss processes the requests without any errors the CPU utilization on the JBoss machine doesnt goes beyond 70%. It is not waiting for I/O, N/W seems perfectly fine and DB machine is also performing optimally. The JVM arguments for JBoss process are tuned for GC, Heap size etc. The h/w on which JBoss is running is powerful enough to handle such load easily.

      I wanted to know what things one should look in to in such a situation? Are there any tools that can point where the bottleneck lies? What JBoss configuration changes one should look in to?

      BTW I am using a slimmed version of JBoss and have done config changes according to the wiki page for performance optimization.

      Thanks,
      -Shrinivas

        • 1. Re: Identifying performance bottlenecks
          otaylor

          I think I'm missing something about the situation. You say that you have a limited
          workload (2000 requests per second are sent), and that you application processes
          them without error. Why do you then expect 100% (or more) CPU utilization? Is your
          application falling behind the workload?

          • 2. Re: Identifying performance bottlenecks
            jshrinivas

            Yes. The throughput is less than expected. Let me put it this way:

            The load driver can load the JBoss machine with as much load as it can handle. So ideally the CPU utilization on JBoss machine should be close to 100% unless there is any bottleneck. In this particular case even if requests are around 2K the throughput is little over 1K. In essence the App Server is not able to handle all the client requests because it is choking somewhere. I have tried increasing the load from 2K to more than 2K and that doesnt makes any difference on CPU utilization/throughput.

            Thanks,
            -Shrinivas

            • 3. Re: Identifying performance bottlenecks
              otaylor

              Without knowing the architecture of your application it's a little hard to provide guesses as to what the bottleneck could be. It could, for example, be network roundtrips if you are making very large numbers of fast database requests to a remote database machine. It could be thread synchronization bottlenecks, etc.

              A useful rough profiling technique is to get a dump of what all the threads in your application are doing: see http://wiki.jboss.org/wiki/Wiki.jsp?page=StackTrace (and follow the "jmx-console" link if you are using recent JBoss). By examining the stack trace, you should be able to see what your application threads are doing most of the time.

              It's possible as well that you just need to increase parallelism ... for example, if you have only one thread handling requests, then you wouldn't expect to be able to get more than about 70% cpu usage (the other 30% of the time is spent waiting for the database, etc.)

              • 4. Re: Identifying performance bottlenecks
                jshrinivas

                Thanks for your reply. It will certainly be helpful to analyse the thread stack trace. Also which thread are you referring to when you say parallelism. Do you mean multiple JVM instances? As far as design of the application is concerned I dont have much details on it but it exercises many of the current J2EE features , EJB, MDB, JMS, Servlets, JSPs , JDBC etc. I wanted to know if there are configuration things (such as for e.g. maxThreads in jbossweb-tomcat5?.sar/server.xml for HTTP connections, JDBC pool size ) related to EJBs, MDBs, JSPs and other entities of a J2EE stack.

                Thanks again,
                -Shrinivas

                • 5. Re: Identifying performance bottlenecks
                  otaylor

                  I don't mean multiple JVMs ... I primarily mean multiple threads. Almost all thread pool sizes in JBoss are configurable; to figure out which one to increase you need to figure out what is relevant for your app ... you could spend weeks increasing different variables without making a difference. For example, if in the stacktraces you see that you have 100 TP-Processor (Tomcat) threads all waiting for a response from your database, then you might want to update maxThreads. (That is unrealistic... few database configurations would increase throughput with more than 100 simulataneous connections but hopefully you get the idea.)

                  • 6. Re: Identifying performance bottlenecks
                    jshrinivas

                    Thanks for the info. Well I guess then it will be a little tricky to identify the bottlenecks. I will post back if I can figure it out.

                    Thanks,
                    -Shrinivas

                    • 7. Re: Identifying performance bottlenecks
                      jshrinivas

                      Analyzing thread stack traces indeed proved useful. Thanks for your help. I looked at http-0.0.0.0-# threads and most of them were in WAIT/BLOCKED state. All these threads seem to be waiting for JDBC connection. Basically it seems that they are waiting on org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection method.

                      During this I also realized that JBoss XA JDBC pool config is not intelligent (or I might be missing something here). I had specified min-pool-size as 200 and did not specify max-pool-size and it defaulted to 20, which was strange. I am going to post this on JCA forum. Anyways, changing max-pool-size helped a bit and I was able to get another 5% CPU utilized due to that.

                      I also found out that there was a bug report on org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection being slow. I am not sure if it is fixed in 4.0.5-SP1-ejb3.

                      -Shrinivas