11 Replies Latest reply on Aug 23, 2019 7:18 AM by Carlos Oliva

    Wildfly 10.1.0.Final threads stuck in infinite loop

    Brian Reed Newbie

      Hi,

       

      We've been facing issues after recently upgrading to Wildfly 10.1.0.Final with threads randomly getting stuck in an infinite loop while writing JSP responses.

       

      The environment is:

       

      OS: Windows 2008 R2

      JVM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13, mixed mode)

      Java: version 1.8.0_121, vendor Oracle Corporation

      Java Home: C:\Program Files\Java\jdk1.8.0_121\jre

      TLS Enabled

       

      We've found that after a few hours or days of running, one or more threads will get stuck. We haven't yet figured out what the trigger is to reproduce it. However, we've had to restart the service at least once a day to maintain responsiveness and clear the looping threads.

       

      Debugging the threads has shown that the io.undertow.server.HttpServerExchange$WriteDispatchChanell.write() method is repeatedly writing 0 bytes and thus never existing the loop in  io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:566).

       

      I haven't found anything that describes this problem for others yet, though there are similar instances on the input side such as this one [UNDERTOW-282] randomly org.xnio.conduits.AbstractStreamSourceConduit.read() goes into infinite loop - JBoss Issue Track…

       

      Here is an example thread dump for one of the hung threads. This thread has been stuck for 2 days. I've also included a screenshot of a JVisualVM CPU sample for this thread.

       

      "default task-44" #290 prio=5 os_prio=0 tid=0x000000002ad80000 nid=0x1b0a4 runnable [0x000000003439b000]

         java.lang.Thread.State: RUNNABLE

        at org.xnio.conduits.ConduitStreamSinkChannel.awaitWritable(ConduitStreamSinkChannel.java:134)

        at io.undertow.channels.DetachableStreamSinkChannel.awaitWritable(DetachableStreamSinkChannel.java:87)

        at io.undertow.server.HttpServerExchange$WriteDispatchChannel.awaitWritable(HttpServerExchange.java:1962)

        at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:566)

        at io.undertow.servlet.spec.ServletOutputStreamImpl.flushInternal(ServletOutputStreamImpl.java:478)

        at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:227)

        at io.undertow.servlet.spec.ServletPrintWriterDelegate.write(ServletPrintWriterDelegate.java:114)

        at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:256)

        at java.io.PrintWriter.write(PrintWriter.java:426)

        - locked <0x000000070e19d4b8> (a org.apache.jasper.runtime.JspWriterImpl)

        at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:256)

        at java.io.PrintWriter.write(PrintWriter.java:426)

        - locked <0x000000070e19d560> (a org.apache.jasper.runtime.JspWriterImpl)

        at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:106)

        at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:307)

        at java.io.Writer.write(Writer.java:157)

        at org.apache.jasper.runtime.JspWriterImpl.print(JspWriterImpl.java:442)

        at org.apache.struts.taglib.TagUtils.write(TagUtils.java:1231)

        at org.apache.struts.taglib.html.LinkTag.doStartTag(LinkTag.java:351)

        at org.apache.jsp.content.jsp.employee.employee.employeeDetailsBody_jsp._jspx_meth_html_005flink_005f0(employeeDetailsBody_jsp.java:8492)

        at org.apache.jsp.content.jsp.employee.employee.employeeDetailsBody_jsp._jspService(employeeDetailsBody_jsp.java:3581)

        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)

        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:402)

        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:346)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

        at io.undertow.jsp.JspFileHandler.handleRequest(JspFileHandler.java:32)

        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)

        at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToServlet(ServletInitialHandler.java:219)

        at io.undertow.servlet.spec.RequestDispatcherImpl.includeImpl(RequestDispatcherImpl.java:377)

        at io.undertow.servlet.spec.RequestDispatcherImpl.setupIncludeImpl(RequestDispatcherImpl.java:311)

        at io.undertow.servlet.spec.RequestDispatcherImpl.include(RequestDispatcherImpl.java:277)

        at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:907)

        at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:631)

        at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99)

        at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135)

        at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:760)

        at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:892)

        at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)

        at org.apache.jsp.content.jsp.common.templates.MaintenanceLayout_jsp._jspx_meth_tiles_005finsert_005f2(MaintenanceLayout_jsp.java:619)

        at org.apache.jsp.content.jsp.common.templates.MaintenanceLayout_jsp._jspService(MaintenanceLayout_jsp.java:380)

        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)

        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:402)

        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:346)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

        at io.undertow.jsp.JspFileHandler.handleRequest(JspFileHandler.java:32)

        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)

        at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToServlet(ServletInitialHandler.java:219)

        at io.undertow.servlet.spec.RequestDispatcherImpl.includeImpl(RequestDispatcherImpl.java:377)

        at io.undertow.servlet.spec.RequestDispatcherImpl.setupIncludeImpl(RequestDispatcherImpl.java:311)

        at io.undertow.servlet.spec.RequestDispatcherImpl.include(RequestDispatcherImpl.java:277)

        at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:907)

        at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:631)

        at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99)

        at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135)

        at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:760)

        at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:892)

        at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)

        at org.apache.jsp.content.jsp.employee.employee.employeeTemplate_jsp._jspx_meth_tiles_005finsert_005f0(employeeTemplate_jsp.java:157)

        at org.apache.jsp.content.jsp.employee.employee.employeeTemplate_jsp._jspService(employeeTemplate_jsp.java:102)

        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)

        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:402)

        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:346)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

        at io.undertow.jsp.JspFileHandler.handleRequest(JspFileHandler.java:32)

        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)

        at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209)

        at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221)

        at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147)

        at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111)

        at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)

        at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)

        at org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:398)

        at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:318)

        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:241)

        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)

        at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)

        at com.tzaconsulting.ptw.filter.SecurityFilter.doFilter(SecurityFilter.java:132)

        at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

        at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

        at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

        at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

        at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)

        at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

        at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

        at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)

        at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

        at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

        at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

        at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

        at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)

        at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)

        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)

        at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)

        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)

        at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)

        at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)

        at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)

        at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

        at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

        at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

        at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

        at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

        at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)

        at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

        at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)

        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

       

       

         Locked ownable synchronizers:

        - <0x000000070de7a0e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

       

       

      default task-44-1.png

      default task-44-2.png

       

      Any suggestions are appreciated.

        • 1. Re: Wildfly 10.1.0.Final threads stuck in infinite loop
          Jaikiran Pai Master

          There seems to be a bug going around in 10.1.0.Final of WildFly. Similar issues, like the one you report, have been discussed in this forum recently and the solution to all those seems to be to disable HTTP2. Here's the attribute which you need to set to false wildfly/wildfly-undertow_3_1.xsd at 10.1.0.Final · wildfly/wildfly · GitHub
          in your subsystem configuration. See if that helps bypass this issue.

          1 of 1 people found this helpful
          • 2. Re: Wildfly 10.1.0.Final threads stuck in infinite loop
            Brian Reed Newbie

            We've made the change and run for a few days without issue. We'll continue to monitor it, but this appears to have resolved the issue. Thanks!

            • 3. Re: Wildfly 10.1.0.Final threads stuck in infinite loop
              valsaraj viswanathan Master

              In which version of WildFly, the fix is released?

              • 4. Re: Wildfly 10.1.0.Final threads stuck in infinite loop
                Olaf Ziems Newbie

                We have a quite similar issue: When our application running on Wildfly 17.0.1 is queried via HTTPS with HTTP2 enabled, we get the following error:

                 

                2019-07-30 13:31:43,545 WARNING [javax.enterprise.resource.webcontainer.jsf.application] (default task-9:) JSF1064: Unable to find or serve resource, fonts/Inter-Light-BETA.woff, from library, theme_modern_dark-blue.
                2019-07-30 13:31:43,545 WARNING [javax.enterprise.resource.webcontainer.jsf.application] (default task-9:) : java.nio.channels.ClosedChannelException
                        at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:238) [undertow-core-2.0.21.Final.jar:2.0.21.Final]
                        at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:2105) [undertow-core-2.0.21.Final.jar:2.0.21.Final]
                        at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:574) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                        at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:150) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                        at org.omnifaces.servlet.GzipHttpServletResponse$GzipThresholdOutputStream.write(GzipHttpServletResponse.java:210) [omnifaces-1.14.1.jar:1.14.1]
                        at org.omnifaces.io.ResettableBufferedOutputStream.write(ResettableBufferedOutputStream.java:79) [omnifaces-1.14.1.jar:1.14.1]
                        at org.omnifaces.servlet.HttpServletResponseOutputWrapper$1.write(HttpServletResponseOutputWrapper.java:100) [omnifaces-1.14.1.jar:1.14.1]
                        at java.nio.channels.Channels$WritableByteChannelImpl.write(Channels.java:458) [rt.jar:1.8.0_91]
                        at com.sun.faces.application.resource.ResourceHandlerImpl.handleResourceRequest(ResourceHandlerImpl.java:318) [jsf-impl-2.3.9.SP02.jar:2.3.9.SP02]
                        at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:171) [jboss-jsf-api_2.3_spec-2.3.9.SP02.jar:2.3.9.SP02]
                        at org.primefaces.application.resource.PrimeResourceHandler.handleResourceRequest(PrimeResourceHandler.java:96) [primefaces-7.0.jar:7.0]
                        at org.richfaces.resource.ResourceHandlerImpl.handleResourceRequest(ResourceHandlerImpl.java:219) [richfaces-core-4.5.17.Final.jar:4.5.17.Final]
                        at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:171) [jboss-jsf-api_2.3_spec-2.3.9.SP02.jar:2.3.9.SP02]
                        at org.apache.myfaces.custom.captcha.CAPTCHAResourceHandlerWrapper.handleResourceRequest(CAPTCHAResourceHandlerWrapper.java:212) [tomahawk21-1.1.14.jar:1.1.14]
                        at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:171) [jboss-jsf-api_2.3_spec-2.3.9.SP02.jar:2.3.9.SP02]
                        at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:171) [jboss-jsf-api_2.3_spec-2.3.9.SP02.jar:2.3.9.SP02]

                 

                The exception occurs, because the channel for sending the resource to the client is closed. If the error occured, Wildfly does not respond anymore: not on HTTP requests or Linux SIGTERM or anything else.

                 

                Disabling HTTP2 for HTTPS fixes the problem.

                 

                Is there still an issue with HTTP2 for Wildfly 17.0.1?

                1 of 1 people found this helpful
                • 5. Re: Wildfly 10.1.0.Final threads stuck in infinite loop
                  Jaikiran Pai Master

                  ziems1  wrote:

                   

                  We have a quite similar issue: When our application running on Wildfly 17.0.1 is queried via HTTPS with HTTP2 enabled, we get the following error:

                   

                  2019-07-30 13:31:43,545 WARNING [javax.enterprise.resource.webcontainer.jsf.application] (default task-9:) JSF1064: Unable to find or serve resource, fonts/Inter-Light-BETA.woff, from library, theme_modern_dark-blue. 2019-07-30 13:31:43,545 WARNING [javax.enterprise.resource.webcontainer.jsf.application] (default task-9:) : java.nio.channels.ClosedChannelException         at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:238) [undertow-core-2.0.21.Final.jar:2.0.21.Final]         at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:2105) [undertow-core-2.0.21.Final.jar:2.0.21.Final]         at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:574) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]         at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:150) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]         at org.omnifaces.servlet.GzipHttpServletResponse$GzipThresholdOutputStream.write(GzipHttpServletResponse.java:210) [omnifaces-1.14.1.jar:1.14.1]         at org.omnifaces.io.ResettableBufferedOutputStream.write(ResettableBufferedOutputStream.java:79) [omnifaces-1.14.1.jar:1.14.1]         at org.omnifaces.servlet.HttpServletResponseOutputWrapper$1.write(HttpServletResponseOutputWrapper.java:100) [omnifaces-1.14.1.jar:1.14.1]         at java.nio.channels.Channels$WritableByteChannelImpl.write(Channels.java:458) [rt.jar:1.8.0_91]         at com.sun.faces.application.resource.ResourceHandlerImpl.handleResourceRequest(ResourceHandlerImpl.java:318) [jsf-impl-2.3.9.SP02.jar:2.3.9.SP02]         at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:171) [jboss-jsf-api_2.3_spec-2.3.9.SP02.jar:2.3.9.SP02]         at org.primefaces.application.resource.PrimeResourceHandler.handleResourceRequest(PrimeResourceHandler.java:96) [primefaces-7.0.jar:7.0]         at org.richfaces.resource.ResourceHandlerImpl.handleResourceRequest(ResourceHandlerImpl.java:219) [richfaces-core-4.5.17.Final.jar:4.5.17.Final]         at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:171) [jboss-jsf-api_2.3_spec-2.3.9.SP02.jar:2.3.9.SP02]         at org.apache.myfaces.custom.captcha.CAPTCHAResourceHandlerWrapper.handleResourceRequest(CAPTCHAResourceHandlerWrapper.java:212) [tomahawk21-1.1.14.jar:1.1.14]         at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:171) [jboss-jsf-api_2.3_spec-2.3.9.SP02.jar:2.3.9.SP02]         at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:171) [jboss-jsf-api_2.3_spec-2.3.9.SP02.jar:2.3.9.SP02]

                   

                  The exception occurs, because the channel for sending the resource to the client is closed.

                   

                  Disabling HTTP2 for HTTPS fixes the problem.

                   

                  Is there still an issue with HTTP2 for Wildfly 17.0.1?

                  Every time this issue has come up in these forums, JSF has been involved. So this looks very specific to serving JSF backed resources over HTTP2. There was even a JIRA or somekind of a github issue related to this, in some project. I can't seem to locate it now though.

                   

                  If the error occured, Wildfly does not respond anymore: not on HTTP requests or Linux SIGTERM or anything else.

                  This part is interesting. Do you have a reproducible application (preferably in source form that can be built) that you can attach to help narrow this down? Alternatively, you can take a thread dump of this Java process and attach that thread dump to this post, whenever this "hang" occurs.

                  • 6. Re: Wildfly 10.1.0.Final threads stuck in infinite loop
                    Olaf Ziems Newbie

                    I guess the problem lies somewhere in our application setup/design/dependencies and cannot provide a sample application to reproduce this error. I attached two thread dumps taken some minutes apart.

                     

                    Additionally here are some more information:

                    • on Wildfly 12 our application ran without any problems with the same configuration
                    • we are currently migrating from RichFaces to PrimeFaces, so both libraries are loaded together
                    • after sending a HTTPS request to our application from a client browser, a 404 error to "/favicon.ico" (possibly unrelated) and a 500 error (internal server error) along with numerous of the ClosedChannelException occur
                    • there also occurs the following error, perhaps it is related:
                    Caused by: java.lang.IllegalStateException: UT000010: Session is invalid b48QW0qtQIngww-9cpc8rpHHQsQMg0DbGenC4Fvc
                            at io.undertow.server.session.InMemorySessionManager$SessionImpl.removeAttribute(InMemorySessionManager.java:549) [undertow-core-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.servlet.spec.HttpSessionImpl.removeAttribute(HttpSessionImpl.java:183) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            at org.springframework.security.web.context.HttpSessionSecurityContextRepository$SaveToSessionResponseWrapper.saveContext(HttpSessionSecurityContextRepository.java:310) [spring-security-web-3.2.5.RELEASE.jar:3.2.5.RELEASE]
                            at org.springframework.security.web.context.HttpSessionSecurityContextRepository.saveContext(HttpSessionSecurityContextRepository.java:117) [spring-security-web-3.2.5.RELEASE.jar:3.2.5.RELEASE]
                            at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:93) [spring-security-web-3.2.5.RELEASE.jar:3.2.5.RELEASE]
                            at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.5.RELEASE.jar:3.2.5.RELEASE]
                            at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) [spring-security-web-3.2.5.RELEASE.jar:3.2.5.RELEASE]
                            at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.2.5.RELEASE.jar:3.2.5.RELEASE]
                            at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) [spring-web-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                            at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) [spring-web-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                            at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            at com.ocpsoft.pretty.PrettyFilter.doFilter(PrettyFilter.java:145) [prettyfaces-jsf2-3.3.3.jar:]
                            at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:103) [spring-web-5.1.8.RELEASE.jar:5.1.8.RELEASE]
                            at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:227) [undertow-servlet-2.0.21.Final.jar:2.0.21.Final]
                            ... 61 more
                    
                    
                    • at some point, an OutOfMemoryError occurs and the loops end and Wildfly becomes responsive again. This can take 20 minutes or more to happen.
                    • 7. Re: Wildfly 10.1.0.Final threads stuck in infinite loop
                      Carlos Oliva Novice

                      It has been discussed (I lack the reference) that this problem is due to the lack of ALPN support in JDK1.8.  We experienced a similar problem with Wildfly 10.1 so we disabled htt2.  The problem went away  Nevertheless, a solution to this problem could be to add support for ALPN to the boot path of the virtual machine.  Have you tried adding ALPN support to the VM with which Wildfly is running?

                      1 of 1 people found this helpful
                      • 9. Re: Wildfly 10.1.0.Final threads stuck in infinite loop
                        Carlos Oliva Novice

                        Hi Val.

                        I think that Wildfly 9 onwards already has http2 enabled -- supported via Undertow.  Some JDK's lack support for ALNP and that is why people see the errors described in this issue.

                         

                        Please review the information in the following links:

                        Using HTTP2 With Wildfly 9.0.0.Beta1 · JBoss Community

                        Re: Does Wildfly (undertow) support http2?

                         

                        It looks like since Wildfly 9.0 onwards, http2 is supported --  ALPN method.  Older JDK's need some libraries to provide ALPN support.  The links describe the steps to take in order to provide this support for the different versions of the JDK.

                        I wanted to confirm that the problems with http2 did indeed disappear by following these steps.

                        • 10. Re: Wildfly 10.1.0.Final threads stuck in infinite loop
                          Olaf Ziems Newbie

                          ramboid  schrieb:

                           

                          Have you tried adding ALPN support to the VM with which Wildfly is running?

                          I tested adding alpn-boot-8.1.7.v20160121.jar for Oracle JDK 1.8.0_91 (on WF17.0.1, Firefox 68.0.2) but still get java.nio.channels.ClosedChannelException and an endless loop in Wildfly (see jstacks above) when using HTTPs with enabled HTTP/2.

                           

                          I also tested with AdoptOpenJDK (build 11.0.4+11), but did not find an ALPN version for that (Chapter 15. Application Layer Protocol Negotiation (ALPN) )

                          • 11. Re: Wildfly 10.1.0.Final threads stuck in infinite loop
                            Carlos Oliva Novice

                            Thank you for your feedback Schrieb.  Have you tried to deploy a simple application without Spring in Wildfly? I noticed that you might be using Spring security in your system. In earlier versions of WIldfly, Wildfly 10.1, we never saw that class instantiated anywhere.