jboss as 7.1.2 apr randomly very slow
I'm running a seam 2.2 - richfaces 3.3.3 on a jboss 7.1.2 as server on a single machine, single JVM Ubuntu Linux(10.04).
Everything is running fine but ocasionally (0.01% of requests) some of the requests take an enormous amount of time to complete: 10s, 60s or even 900s. The usual request time (for 99% of requests) is under 500ms.
Worse than this, during the slow requests processing the CPU usage skyrockets to 50% from the usually 5-10%. Garbage collection at the time is very low, or absent. The system is not at all under heavy user load, the problem might apear with only one connected user.
I managed to get a thread dump during the slow requests processing and it seems the time is spent in tomcat native communication:
Thread 12229: (state = IN_NATIVE) - org.apache.tomcat.jni.Socket.sendbb(long, int, int) @bci=0 (Interpreted frame) - org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer() @bci=364, line=854 (Interpreted frame) - org.apache.coyote.http11.InternalAprOutputBuffer$SocketOutputBuffer.doWrite(org.apache.tomcat.util.buf.ByteChunk, org.apache.coyote.Response) @bci=80, line=897 (Compiled frame) - org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(org.apache.tomcat.util.buf.ByteChunk, org.apache.coyote.Response) @bci=97, line=126 (Compiled frame) - org.apache.coyote.http11.InternalAprOutputBuffer.doWrite(org.apache.tomcat.util.buf.ByteChunk, org.apache.coyote.Response) @bci=87, line=557 (Interpreted frame) - org.apache.coyote.Response.doWrite(org.apache.tomcat.util.buf.ByteChunk) @bci=6, line=594 (Interpreted frame) - org.apache.catalina.connector.OutputBuffer.realWriteBytes(byte[], int, int) @bci=38, line=398 (Interpreted frame) - org.apache.tomcat.util.buf.ByteChunk.flushBuffer() @bci=71, line=449 (Compiled frame) - org.apache.catalina.connector.OutputBuffer.realWriteChars(char[], int, int) @bci=62, line=473 (Compiled frame) - org.apache.tomcat.util.buf.CharChunk.append(char[], int, int) @bci=224, line=305 (Interpreted frame) - org.apache.catalina.connector.OutputBuffer.write(char[], int, int) @bci=15, line=505 (Interpreted frame) - org.apache.catalina.connector.CoyoteWriter.write(char[], int, int) @bci=15, line=143 (Interpreted frame) - org.ajax4jsf.io.FastBufferReader.writeTo(java.io.Writer) @bci=83, line=134 (Compiled frame) - org.ajax4jsf.io.parser.FastHtmlParser.parse(java.io.Reader, java.io.Writer) @bci=553, line=169 (Compiled frame) - org.ajax4jsf.io.parser.FastHtmlParser.parseHtml(java.io.Reader, java.io.Writer) @bci=3, line=230 (Interpreted frame) - org.ajax4jsf.webapp.FilterServletResponseWrapper.parseContent(java.io.Writer, org.ajax4jsf.webapp.HtmlParser) @bci=23, line=596 (Interpreted frame) - org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(javax.servlet.FilterChain, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=996, line=399 (Interpreted frame) - org.ajax4jsf.webapp.BaseFilter.handleRequest(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, javax.servlet.FilterChain) @bci=31, line=290 (Interpreted frame) - org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, javax.servlet.FilterChain) @bci=301, line=388 (Interpreted frame) - org.ajax4jsf.webapp.BaseFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=352, line=515 (Interpreted frame) - org.jboss.seam.web.Ajax4jsfFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=72, line=56 (Interpreted frame) - org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=72, line=69 (Interpreted frame) - org.jboss.seam.servlet.SeamFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=59, line=158 (Interpreted frame) - org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=141, line=280 (Interpreted frame) - org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=248 (Interpreted frame) - com.gradientis.sombrero.filter.SecuringClientTokenFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=61, line=21 (Interpreted frame) - org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=141, line=280 (Interpreted frame) - org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=248 (Interpreted frame) - org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=684, line=275 (Interpreted frame) - org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=200, line=161 (Interpreted frame) - org.apache.catalina.authenticator.AuthenticatorBase.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=311, line=397 (Interpreted frame) - org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=9, line=50 (Interpreted frame) - org.apache.catalina.valves.AccessLogValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=24, line=567 (Compiled frame) - org.jboss.as.web.security.SecurityContextAssociationValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=430, line=153 (Interpreted frame) - org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=233, line=155 (Compiled frame) - org.apache.catalina.valves.ErrorReportValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=6, line=102 (Interpreted frame) - org.apache.catalina.core.StandardEngineValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response) @bci=42, line=109 (Interpreted frame) - org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request, org.apache.coyote.Response) @bci=258, line=368 (Interpreted frame) - org.apache.coyote.http11.Http11AprProcessor.process(long) @bci=375, line=897 (Compiled frame) - org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(long) @bci=22, line=634 (Interpreted frame) - org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run() @bci=11, line=2516 (Interpreted frame) - org.jboss.threads.SimpleDirectExecutor.execute(java.lang.Runnable) @bci=1, line=33 (Interpreted frame) - org.jboss.threads.QueueExecutor.runTask(java.lang.Runnable) @bci=9, line=801 (Interpreted frame) - org.jboss.threads.QueueExecutor.access$100(org.jboss.threads.QueueExecutor, java.lang.Runnable) @bci=2, line=45 (Interpreted frame) - org.jboss.threads.QueueExecutor$Worker.run() @bci=294, line=842 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) - org.jboss.threads.JBossThread.run() @bci=10, line=122 (Interpreted frame)
The https connector configuration I'm using is the following:
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="true"> <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true" max-connections="100" executor="https-executor"> <ssl password="***" certificate-key-file="/some_path" certificate-file="/some_path"/> </connector> ...
I googled around for a similar problem and found something similar:
But no answer
Did I forgot to set an important apr connector configuration parameter? Or could it be the version of the apr shipped with jboss 7.1.2 (see the above links)? Or could it be something else?
