-
1. Re: Identifying performance bottlenecks
otaylor Nov 29, 2006 12:01 PM (in response to jshrinivas)I think I'm missing something about the situation. You say that you have a limited
workload (2000 requests per second are sent), and that you application processes
them without error. Why do you then expect 100% (or more) CPU utilization? Is your
application falling behind the workload? -
2. Re: Identifying performance bottlenecks
jshrinivas Nov 29, 2006 12:26 PM (in response to jshrinivas)Yes. The throughput is less than expected. Let me put it this way:
The load driver can load the JBoss machine with as much load as it can handle. So ideally the CPU utilization on JBoss machine should be close to 100% unless there is any bottleneck. In this particular case even if requests are around 2K the throughput is little over 1K. In essence the App Server is not able to handle all the client requests because it is choking somewhere. I have tried increasing the load from 2K to more than 2K and that doesnt makes any difference on CPU utilization/throughput.
Thanks,
-Shrinivas -
3. Re: Identifying performance bottlenecks
otaylor Nov 29, 2006 1:02 PM (in response to jshrinivas)Without knowing the architecture of your application it's a little hard to provide guesses as to what the bottleneck could be. It could, for example, be network roundtrips if you are making very large numbers of fast database requests to a remote database machine. It could be thread synchronization bottlenecks, etc.
A useful rough profiling technique is to get a dump of what all the threads in your application are doing: see http://wiki.jboss.org/wiki/Wiki.jsp?page=StackTrace (and follow the "jmx-console" link if you are using recent JBoss). By examining the stack trace, you should be able to see what your application threads are doing most of the time.
It's possible as well that you just need to increase parallelism ... for example, if you have only one thread handling requests, then you wouldn't expect to be able to get more than about 70% cpu usage (the other 30% of the time is spent waiting for the database, etc.) -
4. Re: Identifying performance bottlenecks
jshrinivas Nov 29, 2006 4:04 PM (in response to jshrinivas)Thanks for your reply. It will certainly be helpful to analyse the thread stack trace. Also which thread are you referring to when you say parallelism. Do you mean multiple JVM instances? As far as design of the application is concerned I dont have much details on it but it exercises many of the current J2EE features , EJB, MDB, JMS, Servlets, JSPs , JDBC etc. I wanted to know if there are configuration things (such as for e.g. maxThreads in jbossweb-tomcat5?.sar/server.xml for HTTP connections, JDBC pool size ) related to EJBs, MDBs, JSPs and other entities of a J2EE stack.
Thanks again,
-Shrinivas -
5. Re: Identifying performance bottlenecks
otaylor Nov 29, 2006 4:54 PM (in response to jshrinivas)I don't mean multiple JVMs ... I primarily mean multiple threads. Almost all thread pool sizes in JBoss are configurable; to figure out which one to increase you need to figure out what is relevant for your app ... you could spend weeks increasing different variables without making a difference. For example, if in the stacktraces you see that you have 100 TP-Processor (Tomcat) threads all waiting for a response from your database, then you might want to update maxThreads. (That is unrealistic... few database configurations would increase throughput with more than 100 simulataneous connections but hopefully you get the idea.)
-
6. Re: Identifying performance bottlenecks
jshrinivas Nov 29, 2006 5:05 PM (in response to jshrinivas)Thanks for the info. Well I guess then it will be a little tricky to identify the bottlenecks. I will post back if I can figure it out.
Thanks,
-Shrinivas -
7. Re: Identifying performance bottlenecks
jshrinivas Nov 30, 2006 3:07 PM (in response to jshrinivas)Analyzing thread stack traces indeed proved useful. Thanks for your help. I looked at http-0.0.0.0-# threads and most of them were in WAIT/BLOCKED state. All these threads seem to be waiting for JDBC connection. Basically it seems that they are waiting on org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection method.
During this I also realized that JBoss XA JDBC pool config is not intelligent (or I might be missing something here). I had specified min-pool-size as 200 and did not specify max-pool-size and it defaulted to 20, which was strange. I am going to post this on JCA forum. Anyways, changing max-pool-size helped a bit and I was able to get another 5% CPU utilized due to that.
I also found out that there was a bug report on org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection being slow. I am not sure if it is fixed in 4.0.5-SP1-ejb3.
-Shrinivas