Wildfly 10.1.0.Final threads stuck in infinite loop
breed Mar 27, 2017 10:49 AMHi,
We've been facing issues after recently upgrading to Wildfly 10.1.0.Final with threads randomly getting stuck in an infinite loop while writing JSP responses.
The environment is:
OS: Windows 2008 R2
JVM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13, mixed mode)
Java: version 1.8.0_121, vendor Oracle Corporation
Java Home: C:\Program Files\Java\jdk1.8.0_121\jre
TLS Enabled
We've found that after a few hours or days of running, one or more threads will get stuck. We haven't yet figured out what the trigger is to reproduce it. However, we've had to restart the service at least once a day to maintain responsiveness and clear the looping threads.
Debugging the threads has shown that the io.undertow.server.HttpServerExchange$WriteDispatchChanell.write() method is repeatedly writing 0 bytes and thus never existing the loop in io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:566).
I haven't found anything that describes this problem for others yet, though there are similar instances on the input side such as this one [UNDERTOW-282] randomly org.xnio.conduits.AbstractStreamSourceConduit.read() goes into infinite loop - JBoss Issue Track…
Here is an example thread dump for one of the hung threads. This thread has been stuck for 2 days. I've also included a screenshot of a JVisualVM CPU sample for this thread.
"default task-44" #290 prio=5 os_prio=0 tid=0x000000002ad80000 nid=0x1b0a4 runnable [0x000000003439b000]
java.lang.Thread.State: RUNNABLE
at org.xnio.conduits.ConduitStreamSinkChannel.awaitWritable(ConduitStreamSinkChannel.java:134)
at io.undertow.channels.DetachableStreamSinkChannel.awaitWritable(DetachableStreamSinkChannel.java:87)
at io.undertow.server.HttpServerExchange$WriteDispatchChannel.awaitWritable(HttpServerExchange.java:1962)
at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:566)
at io.undertow.servlet.spec.ServletOutputStreamImpl.flushInternal(ServletOutputStreamImpl.java:478)
at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:227)
at io.undertow.servlet.spec.ServletPrintWriterDelegate.write(ServletPrintWriterDelegate.java:114)
at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:256)
at java.io.PrintWriter.write(PrintWriter.java:426)
- locked <0x000000070e19d4b8> (a org.apache.jasper.runtime.JspWriterImpl)
at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:256)
at java.io.PrintWriter.write(PrintWriter.java:426)
- locked <0x000000070e19d560> (a org.apache.jasper.runtime.JspWriterImpl)
at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:106)
at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:307)
at java.io.Writer.write(Writer.java:157)
at org.apache.jasper.runtime.JspWriterImpl.print(JspWriterImpl.java:442)
at org.apache.struts.taglib.TagUtils.write(TagUtils.java:1231)
at org.apache.struts.taglib.html.LinkTag.doStartTag(LinkTag.java:351)
at org.apache.jsp.content.jsp.employee.employee.employeeDetailsBody_jsp._jspx_meth_html_005flink_005f0(employeeDetailsBody_jsp.java:8492)
at org.apache.jsp.content.jsp.employee.employee.employeeDetailsBody_jsp._jspService(employeeDetailsBody_jsp.java:3581)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:402)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:346)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.jsp.JspFileHandler.handleRequest(JspFileHandler.java:32)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)
at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToServlet(ServletInitialHandler.java:219)
at io.undertow.servlet.spec.RequestDispatcherImpl.includeImpl(RequestDispatcherImpl.java:377)
at io.undertow.servlet.spec.RequestDispatcherImpl.setupIncludeImpl(RequestDispatcherImpl.java:311)
at io.undertow.servlet.spec.RequestDispatcherImpl.include(RequestDispatcherImpl.java:277)
at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:907)
at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:631)
at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99)
at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135)
at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:760)
at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:892)
at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)
at org.apache.jsp.content.jsp.common.templates.MaintenanceLayout_jsp._jspx_meth_tiles_005finsert_005f2(MaintenanceLayout_jsp.java:619)
at org.apache.jsp.content.jsp.common.templates.MaintenanceLayout_jsp._jspService(MaintenanceLayout_jsp.java:380)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:402)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:346)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.jsp.JspFileHandler.handleRequest(JspFileHandler.java:32)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)
at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToServlet(ServletInitialHandler.java:219)
at io.undertow.servlet.spec.RequestDispatcherImpl.includeImpl(RequestDispatcherImpl.java:377)
at io.undertow.servlet.spec.RequestDispatcherImpl.setupIncludeImpl(RequestDispatcherImpl.java:311)
at io.undertow.servlet.spec.RequestDispatcherImpl.include(RequestDispatcherImpl.java:277)
at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:907)
at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:631)
at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99)
at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135)
at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:760)
at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:892)
at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)
at org.apache.jsp.content.jsp.employee.employee.employeeTemplate_jsp._jspx_meth_tiles_005finsert_005f0(employeeTemplate_jsp.java:157)
at org.apache.jsp.content.jsp.employee.employee.employeeTemplate_jsp._jspService(employeeTemplate_jsp.java:102)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:402)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:346)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.jsp.JspFileHandler.handleRequest(JspFileHandler.java:32)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)
at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209)
at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221)
at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147)
at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111)
at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)
at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)
at org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:398)
at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:318)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:241)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at com.tzaconsulting.ptw.filter.SecurityFilter.doFilter(SecurityFilter.java:132)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)
at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x000000070de7a0e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Any suggestions are appreciated.