High CPU Utilization

[walanj.amol]

Hi ,

We have started our application with following settings.

$JAVA_OPTS -Xmx4096m -Xms4096m -Xmn2g -Xss128k -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -XX:MaxPermSize=512m -XX:PermSize=256m

We are sending the direct http requst to our application.

It causing the CPU Utilization in the range of 80-85 %.

When we tried jst returning back from the Method with Null response Our CPU Utilization is in the range of 45-50%

We are using the axis2 webservice in it.

Hardware Configuration
Application Node	No of CPU	8
	CPU Type	Intel(R) Xeon(R) CPU E5530 @ 2.40GHz, cache size 8192 KB
	Memory	8 GB
	Network Card	100 mbps Ethernet
Database server	No of CPU	8
	CPU Type	Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, cache size 12228 KB
	Memory	160 GB
	Network Card	100 mbps Ethernet

So currently we are struggling to reduce the CPU usage.

[wdfink]

What threads of the JBoss-Java process consume the CPU?

Or is it during runtime (not the first call) and the GC is active?

[walanj.amol]

Hi , Thanks for quick reply.

Its during the runtime when we fired the bulk orders. (Client with 20 threads)

What I had observed from jconsole is , the tomcat threads (http-0.0.0.0-8080-*) are taking more time to complete.

I had tried to find the top 5 threads consuming the CPU during this run using the shell script.

The Partial O/P is as follows :

PID USER	PR  NI  VIRT  RES  SHR S %CPU %MEM	TIME+  COMMAND
11935 sis	15   0 6342m 3.7g  11m S 29.5 47.0   0:10.37 java
11681 sis	15   0 6342m 3.7g  11m R 25.6 47.0   0:27.65 java
11662 sis	15   0 6342m 3.7g  11m S 23.6 47.0   0:31.89 java
11682 sis	15   0 6342m 3.7g  11m S 21.6 47.0   0:31.34 java
11663 sis	15   0 6342m 3.7g  11m S 21.6 47.0   0:33.84 java

Pasting here the thread Dump for one of the PID.The others dump also shows the same dump.

========> Java LWP: 11681 - Native Thread ID=2da1

"http-0.0.0.0-8080-15" daemon prio=10 tid=0x00002aaac0877000 nid=0x2da1 waiting on condition [0x0000000042e7f000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x0000000746df1920> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)

        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)

        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)

        at org.jboss.security.plugins.JBossAuthorizationManager.internalAuthorization(JBossAuthorizationManager.java:470)

        at org.jboss.security.plugins.JBossAuthorizationManager.authorize(JBossAuthorizationManager.java:125)

        at org.jboss.security.plugins.javaee.WebAuthorizationHelper.hasUserDataPermission(WebAuthorizationHelper.java:201)

        at org.jboss.web.tomcat.security.JBossWebRealm.hasUserDataPermission(JBossWebRealm.java:643)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)

        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)

        at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)

        at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)

        at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:905)

        at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:592)

        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2036)

        at java.lang.Thread.run(Thread.java:662)

Thanks,

Amol

[walanj.amol]

Hi had found one more information related to this.
The CPU Utilization is depend on the maxThreads configured in server.xml for http connector.

I can see the almost linear increase in CPU Utilzation and TPS with the maxThread.

When maxThread reaches approx. 75% of the threads from client its shows the CPU utilization in the range of 80-85%

When I had configured the maxThread=5 , I can see the CPU Utilization in the range of 45-50 % , with drop in TPS almost 40-45%.

When I had configured the maxThread=8 , I can see the CPU Utilization in the range of 70-75 %, with drop in TPS almost 30-35%.

Also I tried to use the executor in server.xml.
But there I got the same result as that of initial configuration.

Thanks,

Amol

[wdfink]

I don't catch it, I suppose I did not understand your scenario.

Do you deploy a WAR application and fire bulk operations with 20 threads parallel.

The server has maxThreads=5 (8), which will put most of the client threads in a wait state.

When do you reach the CPU utilization and what TPS mean?

And what do you want to achieve? More througput or less CPU?

[peterj]

Do you by any chance work with Rajesh Gupta? See: https://community.jboss.org/message/732025#732025

Alos, the more load you put on the system the more threads that will be used to handle the load. This explains the CPU utilization increase under higher load (or when the number of threads increases). Usually there is a "sweet spot" for the number of threads as compared the to number of CPUs you have. Then there is also processor affinity which can play an importanmt role - somtimes the JVM behaves better if it is limited to 2 or 4 CPU even if you have 8 available.

[walanj.amol]

My goal is to gain high TPS with avg. response time below 25ms and low CPU Utilization.
My client is a normal HttpClient written in Java.

Using Thread Pool Executor I manged my client side threads.

Peter ,

I didn't worked with Rajesh. The issue you mentioned also looks similar to mine.

But here you mentioned by increasing the threads the more load on CPU.

what I had found from my testing is there is 1-2 % increase when I increase almost 10 threads from the client.

But the CPU is largely depend on the maxThreads of the http connector configuration in server.xml under deploy/jbossweb.sar directory.

I had calculated the time taken to complete 1 Query.

It's in the range of 5-15ms

So What I can see that the max CPU is Utilized in Handling connections.

So I want to optimize this connections handling so that to reduce CPU with high TPS.

Thanks,

Amol

[peterj]

To reduce CPU utilization, reduce the load! I don't see how you can expect the CPU utilization to go down or remain steady when you add load. (I assume that when you say to added 10 threads to the client that you are using a load testing tool to simulate load and that those ten threads added additonal load on the system.) In all the tests I have done we have constantly increase then load in steps (such as added 10 more simulated users to the load testing tool) and then see what happens at each phase along the way. We usually increase the load until adding more load doesn't yield any more throughput, or even when throughput goes down. That's when we know we have a bottleneck and we hunt it down (and hunting it down takes performance experrts that look at everything from operating system performance data, to database performance data, to JVM performance data, to applicaiton experts that look for hot spots in the application). But trying to keep CPU utilization low while increasing load, and there is no bottleneck yet, is really a waste of time. And at times when the load gets really high and we reach a bottleneck, if it is really just the CPU (that is, not other tuning will work), then we know we have reached the limit on a single machine and will roll in another machine.

[walanj.amol]

Hi Peter ,

Thanks for reply

I agreed with your comment.
But our issue is we are getting high CPU for the minimal load itself.
I am getting 78-82 % CPU with 12 threads and 80-85 % with 20 threads.
And we supposed to have that much load (Concurrent users) in the production system.

[peterj]

Sounds like you need to profile your code and find out where it is spending its time.