2 Replies Latest reply on Mar 31, 2017 10:30 AM by Brian Reed

    Wildfly 10.1.0.Final threads stuck in infinite loop

    Brian Reed Newbie

      Hi,

       

      We've been facing issues after recently upgrading to Wildfly 10.1.0.Final with threads randomly getting stuck in an infinite loop while writing JSP responses.

       

      The environment is:

       

      OS: Windows 2008 R2

      JVM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13, mixed mode)

      Java: version 1.8.0_121, vendor Oracle Corporation

      Java Home: C:\Program Files\Java\jdk1.8.0_121\jre

      TLS Enabled

       

      We've found that after a few hours or days of running, one or more threads will get stuck. We haven't yet figured out what the trigger is to reproduce it. However, we've had to restart the service at least once a day to maintain responsiveness and clear the looping threads.

       

      Debugging the threads has shown that the io.undertow.server.HttpServerExchange$WriteDispatchChanell.write() method is repeatedly writing 0 bytes and thus never existing the loop in  io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:566).

       

      I haven't found anything that describes this problem for others yet, though there are similar instances on the input side such as this one [UNDERTOW-282] randomly org.xnio.conduits.AbstractStreamSourceConduit.read() goes into infinite loop - JBoss Issue Track…

       

      Here is an example thread dump for one of the hung threads. This thread has been stuck for 2 days. I've also included a screenshot of a JVisualVM CPU sample for this thread.

       

      "default task-44" #290 prio=5 os_prio=0 tid=0x000000002ad80000 nid=0x1b0a4 runnable [0x000000003439b000]

         java.lang.Thread.State: RUNNABLE

        at org.xnio.conduits.ConduitStreamSinkChannel.awaitWritable(ConduitStreamSinkChannel.java:134)

        at io.undertow.channels.DetachableStreamSinkChannel.awaitWritable(DetachableStreamSinkChannel.java:87)

        at io.undertow.server.HttpServerExchange$WriteDispatchChannel.awaitWritable(HttpServerExchange.java:1962)

        at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:566)

        at io.undertow.servlet.spec.ServletOutputStreamImpl.flushInternal(ServletOutputStreamImpl.java:478)

        at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:227)

        at io.undertow.servlet.spec.ServletPrintWriterDelegate.write(ServletPrintWriterDelegate.java:114)

        at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:256)

        at java.io.PrintWriter.write(PrintWriter.java:426)

        - locked <0x000000070e19d4b8> (a org.apache.jasper.runtime.JspWriterImpl)

        at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:256)

        at java.io.PrintWriter.write(PrintWriter.java:426)

        - locked <0x000000070e19d560> (a org.apache.jasper.runtime.JspWriterImpl)

        at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:106)

        at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:307)

        at java.io.Writer.write(Writer.java:157)

        at org.apache.jasper.runtime.JspWriterImpl.print(JspWriterImpl.java:442)

        at org.apache.struts.taglib.TagUtils.write(TagUtils.java:1231)

        at org.apache.struts.taglib.html.LinkTag.doStartTag(LinkTag.java:351)

        at org.apache.jsp.content.jsp.employee.employee.employeeDetailsBody_jsp._jspx_meth_html_005flink_005f0(employeeDetailsBody_jsp.java:8492)

        at org.apache.jsp.content.jsp.employee.employee.employeeDetailsBody_jsp._jspService(employeeDetailsBody_jsp.java:3581)

        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)

        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:402)

        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:346)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

        at io.undertow.jsp.JspFileHandler.handleRequest(JspFileHandler.java:32)

        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)

        at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToServlet(ServletInitialHandler.java:219)

        at io.undertow.servlet.spec.RequestDispatcherImpl.includeImpl(RequestDispatcherImpl.java:377)

        at io.undertow.servlet.spec.RequestDispatcherImpl.setupIncludeImpl(RequestDispatcherImpl.java:311)

        at io.undertow.servlet.spec.RequestDispatcherImpl.include(RequestDispatcherImpl.java:277)

        at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:907)

        at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:631)

        at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99)

        at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135)

        at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:760)

        at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:892)

        at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)

        at org.apache.jsp.content.jsp.common.templates.MaintenanceLayout_jsp._jspx_meth_tiles_005finsert_005f2(MaintenanceLayout_jsp.java:619)

        at org.apache.jsp.content.jsp.common.templates.MaintenanceLayout_jsp._jspService(MaintenanceLayout_jsp.java:380)

        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)

        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:402)

        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:346)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

        at io.undertow.jsp.JspFileHandler.handleRequest(JspFileHandler.java:32)

        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)

        at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToServlet(ServletInitialHandler.java:219)

        at io.undertow.servlet.spec.RequestDispatcherImpl.includeImpl(RequestDispatcherImpl.java:377)

        at io.undertow.servlet.spec.RequestDispatcherImpl.setupIncludeImpl(RequestDispatcherImpl.java:311)

        at io.undertow.servlet.spec.RequestDispatcherImpl.include(RequestDispatcherImpl.java:277)

        at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:907)

        at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:631)

        at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99)

        at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135)

        at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:760)

        at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:892)

        at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)

        at org.apache.jsp.content.jsp.employee.employee.employeeTemplate_jsp._jspx_meth_tiles_005finsert_005f0(employeeTemplate_jsp.java:157)

        at org.apache.jsp.content.jsp.employee.employee.employeeTemplate_jsp._jspService(employeeTemplate_jsp.java:102)

        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)

        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:402)

        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:346)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

        at io.undertow.jsp.JspFileHandler.handleRequest(JspFileHandler.java:32)

        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)

        at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209)

        at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221)

        at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147)

        at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111)

        at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)

        at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)

        at org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:398)

        at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:318)

        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:241)

        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)

        at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)

        at com.tzaconsulting.ptw.filter.SecurityFilter.doFilter(SecurityFilter.java:132)

        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

        at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

        at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

        at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)

        at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

        at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

        at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)

        at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

        at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

        at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

        at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)

        at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)

        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)

        at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)

        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)

        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)

        at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)

        at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)

        at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

        at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

        at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

        at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

        at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

        at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)

        at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

        at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)

        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

       

       

         Locked ownable synchronizers:

        - <0x000000070de7a0e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

       

       

      default task-44-1.png

      default task-44-2.png

       

      Any suggestions are appreciated.