High CPU Utilization in JBOSS 5.0.0 GA

[mmuthu]

We have been trying to upgrade our services to JBOSS 5.0 GA for a while. We were able to fix many functional issues. However when we take this to our performance environment what we are having many surpises. The following is the current status of this project and we are stuck here,

Initially we ran with JBOSS 5.0 GA with JRE 1.5. This gave us very poor performance. The following are the performance results. The results are averaged out,

TPS is 6.4
CPU utilization is around 80%
Memory utilization is around 1.3 GB

Later we upgraded to use JBOSS 5.0 GA with JRE 1.6 and we got a better TPS (~8.4) but the remaining variables are same as before.

We took thread dumps many times and there were no blocking threads. However we can see that there are many threads in WAITING state in the following condition,

at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
at java.lang.Thread.run(Unknown Source)

We tried to bring up the services with Apache Portable Runtime(APR) and we are not successful here. The service start up always fails with the following exception trace,

13:11:49,720 INFO [TransactionManagerService] Starting transaction recovery manager
13:11:50,537 ERROR [Http11AprProtocol] Error initializing endpoint
java.lang.Exception: Invalid Server SSL Protocol
at org.apache.tomcat.jni.SSLContext.make(Native Method)
at org.apache.tomcat.util.net.AprEndpoint.init(AprEndpoint.java:656)
at org.apache.coyote.http11.Http11AprProtocol.init(Http11AprProtocol.java:107)
at org.apache.catalina.connector.Connector.initialize(Connector.java:1029)
at org.apache.catalina.core.StandardService.initialize(StandardService.java:683)
at org.apache.catalina.core.StandardService.init(StandardService.java:695)
at org.apache.catalina.core.StandardService.start(StandardService.java:510)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:736)
at org.apache.catalina.startup.Catalina.start(Catalina.java:572)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:297)
at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:164)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
at org.jboss.web.tomcat.service.deployers.TomcatService.startService(TomcatService.java:312)
at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376)

I'm pretty confident that the certificate and configuration is correct. Also one important thing is that the APR is getting used with HTTP with out any issue. It gives me the above problem only with HTTPS enabled connector. The following is our server.xml configuration,

<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" />
<Connector SSLEngine="on" port="8443" SSLEnabled="true" address="${jboss.bind.address}"
 maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
 scheme="https" secure="true" clientAuth="false"
 emptySessionPath="true"
 SSLPassword="abc123" sslProtocol="TLSv1"
 SSLCertificateFile="/opt/jboss-5.0.0.GA/server/default/conf/server.crt"
 SSLCertificateKeyFile="/opt/jboss-5.0.0.GA/server/default/conf/server.key"/>

I thought we should try it with NIO connector however the support for this connector is not available in JBOSS 5.0. Not sure why?

To prove this point i download the JBOSS web module and able to bring up the services with APR with out any issue. I have used JBOSS WEB 2.1.2 GA release. (PS: I anyway face different issue atleast i'm not getting Invalid Server SSL protocol). The

APR version that i used both case is

jboss-native-2.0.6-linux2-x86-ssl

Please note that our log level is reasonable and hence there is no point of thinking about too many logging consume CPU

The JVM parameter that we use is like below,

Xms768m -Xmx1536m
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-verbose:gc
-server
-Xloggc:/var/log/jvm.log
-Djava.awt.headless=true
-XX:MaxPermSize=256m
-Djava.net.preferIPv4Stack=true

We have got a good box configuration. It is running with 4 CPU and 16 GB of RAM and we run only two instance our application. I do not think we need to check anything from this point.

It is pretty much look like we have a problem with tomcat (or JBOSS web). Also currently we are trying to do the following test,

1. Modify the following "net.ipv4.tcp_fin_timeout=10", "net.ipv4.tcp_keepalive_intvl=15", "net.ipv4.tcp_keepalive_probes" and Perform the load test again.
2. Bring up services using JRE 1.6.0_14
3. Run Tomcat (6.0.20) separately as webserver. Disable JBOSS web and start the load test

We can understand that JBOSS consumes additional 400 MB of memory in 5.0. Refer https://jira.jboss.org/jira/browse/JBAS-6549.

With respect to High CPU utilization I think that we are pretty much stuck here. If there are any option that we should try please let me know. Also please let me know if you need any more details to provide suggestions.

With JBOSS 423 we used to get 8.4 TPS, Memory usage is around 800~MB and the CPU utilization is around 30 - 40%

Thanks,
Muthu

[jaikiran]

CPU utilization is around 80%

When are you computing this utilization? Is it during deployment of your application? Or is it when your application is running? If it's the latter then please provide more information about what kind of operations the application does during this period.

[manikganguli]

Faced the same issue . From thread dump we can also see the most cpu consuming threads are SSL threads . isthere any way to avoid this as well :
Thread[http-0.0.0.0-8743-1,5,jboss]


at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:789)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:601)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Unknown Source)

Regards
-Manik

[mmuthu]

Jai,

We generally run our load for 1hr and take the output given by load runner. Well, it is happening while application running under load. The deployment looks solid. Regarding the operation that we perform with our load test is something like below,

login to application, browse through various pages and logout from the application.

I think it is pretty complex for me to define each operations. However if you would like to see the sample view our application which is running using JBOSS 423 is available here,

https://moneycenter.yodlee.com

[jaikiran]

"Manik Ganguli" wrote:


Faced the same issue . From thread dump we can also see the most cpu consuming threads are SSL threads .

I'm not an expert at this. If that's what you have narrowed down the issue to, then let's see if someone with more experience in this area can help.