6 Replies Latest reply on Oct 17, 2017 8:03 AM by Stan Golemanov

    worker tasks hangs on connection reset on wildfly 10.1.0.Final

    Stan Golemanov Newbie

      Hi,

       

      our setup is as follows

       

      Linux + Java 1.8.0_77-b03 + Wildfly10.1.0 + Spring MVC 4.01 + Thymeleaf 3.0.8 ( have the same issue with  Thymeleaf 3.0.0 or/and Wildfly 10.0.0 and 8.2.0 )

       

      We run multiple web sites projects with this setup on different machines.

      One of our clients, reports that occasionally their web site stops responding and needs a restart.

      After some investigations, we found that all of the worker threads ends( 2 core processor, so by default 32 threads) in permanent RUNNABLE state.

      This at first seemed that it happens randomly, but with some more investigations we found a way to hang a thread almost every time

       

      it requires to use wget on some of the urls on the site and stops it ( ctrl+c) before it end naturally ( can be achieved with killing the browser too )

      it looks like this

      wget -r -T 60  http://192.168.10.183:8081/bg/test2/

      --2017-10-06 16:08:58--  http://192.168.10.183:8081/bg/test2/

      Connecting to 192.168.10.183:8081... connected.

      HTTP request sent, awaiting response...

      Ctrl+c here

       

      using JProfilier it looks like Gyazo - 3ddf639bd256a18637f982086304f6b2.png  as task 2 never ends RUNNABLE state

      thread dumping shows that that thread locks

       

      dump just after wget command

      "default task-2" #128 prio=5 os_prio=0 tid=0x0000000002a8a800 nid=0x407a runnable [0x00007fd967ffb000]

         java.lang.Thread.State: RUNNABLE

      at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:150)

      at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:296)

      at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:300)

      at io.undertow.servlet.spec.ServletPrintWriterDelegate.write(ServletPrintWriterDelegate.java:129)

      at org.thymeleaf.engine.OpenElementTag.write(OpenElementTag.java:148)

      at org.thymeleaf.engine.OutputTemplateHandler.handleOpenElement(OutputTemplateHandler.java:142)

      at org.thymeleaf.engine.ProcessorTemplateHandler.handleOpenElement(ProcessorTemplateHandler.java:1576)

      at org.thymeleaf.engine.OpenElementTag.beHandled(OpenElementTag.java:205)

      at org.thymeleaf.engine.Model.process(Model.java:282)

      at org.thymeleaf.engine.Model.process(Model.java:290)

      at org.thymeleaf.engine.IteratedGatheringModelProcessable.processIterationModel(IteratedGatheringModelProcessable.java:367)

      at org.thymeleaf.engine.IteratedGatheringModelProcessable.process(IteratedGatheringModelProcessable.java:221)

      at org.thymeleaf.engine.ProcessorTemplateHandler.handleCloseElement(ProcessorTemplateHandler.java:1640)

      at org.thymeleaf.engine.CloseElementTag.beHandled(CloseElementTag.java:139)

      at org.thymeleaf.engine.Model.process(Model.java:282)

      at org.thymeleaf.engine.ProcessorTemplateHandler.handleOpenElement(ProcessorTemplateHandler.java:1587)

      at org.thymeleaf.engine.OpenElementTag.beHandled(OpenElementTag.java:205)

      at org.thymeleaf.engine.Model.process(Model.java:282)

      at org.thymeleaf.engine.Model.process(Model.java:290)

      at org.thymeleaf.engine.GatheringModelProcessable.process(GatheringModelProcessable.java:78)

      at org.thymeleaf.engine.ProcessorTemplateHandler.handleStandaloneElement(ProcessorTemplateHandler.java:993)

      at org.thymeleaf.engine.StandaloneElementTag.beHandled(StandaloneElementTag.java:228)

      at org.thymeleaf.engine.Model.process(Model.java:282)

      at org.thymeleaf.engine.Model.process(Model.java:290)

      at org.thymeleaf.engine.IteratedGatheringModelProcessable.processIterationModel(IteratedGatheringModelProcessable.java:367)

      at org.thymeleaf.engine.IteratedGatheringModelProcessable.process(IteratedGatheringModelProcessable.java:221)

      at org.thymeleaf.engine.ProcessorTemplateHandler.handleCloseElement(ProcessorTemplateHandler.java:1640)

      at org.thymeleaf.engine.CloseElementTag.beHandled(CloseElementTag.java:139)

      at org.thymeleaf.engine.TemplateModel.process(TemplateModel.java:136)

      at org.thymeleaf.engine.TemplateManager.parseAndProcess(TemplateManager.java:592)

      at org.thymeleaf.TemplateEngine.process(TemplateEngine.java:1098)

      at org.thymeleaf.TemplateEngine.process(TemplateEngine.java:1072)

      at org.thymeleaf.spring4.view.ThymeleafView.renderFragment(ThymeleafView.java:353)

      at org.thymeleaf.spring4.view.ThymeleafView.render(ThymeleafView.java:191)

      at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1225)

      at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1012)

      at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)

      at unisoftbg.web.app.AppDispatchServlet.doDispatch(AppDispatchServlet.java:63)

      at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:876)

      at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)

      at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)

      at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

      at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)

      at unisoftbg.web.filter.HeaderFilter.doFilter(HeaderFilter.java:91)

      at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)

      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

      at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)

      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)

      at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)

      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

      at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

      at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

      at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

      at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

      at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

      at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

      at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

      at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

      at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

      at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)

      at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

      at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

      at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

      at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

      at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

      at java.lang.Thread.run(Thread.java:745)

       

       

         Locked ownable synchronizers:

      - <0x00000006ef6c3150> (a java.util.concurrent.ThreadPoolExecutor$Worker)

       

      dump after some time

      "default task-2" #128 prio=5 os_prio=0 tid=0x0000000002a8a800 nid=0x407a runnable [0x00007fd967ffe000]

         java.lang.Thread.State: RUNNABLE

      at sun.nio.cs.UTF_8$Encoder.encodeBufferLoop(UTF_8.java:645)

      at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:693)

      at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:579)

      at io.undertow.servlet.spec.ServletPrintWriter.close(ServletPrintWriter.java:104)

      at io.undertow.servlet.spec.ServletPrintWriterDelegate.close(ServletPrintWriterDelegate.java:99)

      at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:470)

      at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:560)

      at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:331)

      at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

      at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

      at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

      at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

      at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

      at java.lang.Thread.run(Thread.java:745)

       

       

         Locked ownable synchronizers:

      - <0x00000006ef6c3150> (a java.util.concurrent.ThreadPoolExecutor$Worker)

       

      It's worth mentioning that there are no exceptions in the server.log or in the console.

      Also with wget we manage to reproduce the problem only when we reset a connection to url which serves Thymeleaf template file with th:include in it.

      We realize that this may be a thymeleaf issue but we can't pinpoint the exact problem.

       

      Any directions for solving the problem will be much appreciated.

       

      Thanks

        • 1. Re: worker tasks hangs on connection reset on wildfly 10.1.0.Final
          jaikiran pai Master

          Any chance you can try what's been suggested here Wildfly 10.1.0.Final threads stuck in infinite loop and see if this is related to your issue.

          • 2. Re: worker tasks hangs on connection reset on wildfly 10.1.0.Final
            Stan Golemanov Newbie

            i saw that, but http2 is not enabled(Gyazo - 58b3d6b17752100a487b499944e619ef.png ), also the problem is the same in 8.2 which does not support http2

             

             

            thanks

            • 3. Re: worker tasks hangs on connection reset on wildfly 10.1.0.Final
              Stan Golemanov Newbie

              some more info

               

              while thread dumping we managed to see this

               

              sun.nio.cs.UTF_8$Encoder.overflow(java.nio.CharBuffer, int) (line: 577)

              sun.nio.cs.UTF_8$Encoder.encodeBufferLoop(java.nio.CharBuffer, java.nio.ByteBuffer) (line: 654)

              sun.nio.cs.UTF_8$Encoder.encodeLoop(java.nio.CharBuffer, java.nio.ByteBuffer) (line: 693)

              java.nio.charset.CharsetEncoder.encode(java.nio.CharBuffer, java.nio.ByteBuffer, boolean) (line: 579)

              io.undertow.servlet.spec.ServletPrintWriter.close() (line: 104)

              io.undertow.servlet.spec.ServletPrintWriterDelegate.close() (line: 99)

              io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter() (line: 491)

              io.undertow.servlet.spec.HttpServletResponseImpl.responseDone() (line: 581)

              io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(io.undertow.server.HttpServerExchange, io.undertow.servlet.handlers.ServletChain, io.undertow.servlet.handlers.ServletRequestContext, javax.servlet.ServletRequest, javax.servlet.ServletResponse) (line: 308)

              io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(io.undertow.server.HttpServerExchange, io.undertow.servlet.handlers.ServletRequestContext, io.undertow.servlet.handlers.ServletChain, javax.servlet.DispatcherType) (line: 247)

              io.undertow.servlet.handlers.ServletInitialHandler.access$000(io.undertow.servlet.handlers.ServletInitialHandler, io.undertow.server.HttpServerExchange, io.undertow.servlet.handlers.ServletRequestContext, io.undertow.servlet.handlers.ServletChain, javax.servlet.DispatcherType) (line: 76)

              io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(io.undertow.server.HttpServerExchange) (line: 166)

              io.undertow.server.Connectors.executeRootHandler(io.undertow.server.HttpHandler, io.undertow.server.HttpServerExchange) (line: 197)

              io.undertow.server.HttpServerExchange$1.run() (line: 759)

              java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) (line: 1142)

              java.util.concurrent.ThreadPoolExecutor$Worker.run() (line: 617)

              java.lang.Thread.run() (line: 745)

               

              which lead us to

              https://github.com/undertow-io/undertow/blob/master/servlet/src/main/java/io/undertow/servlet/spec/ServletPrintWriter.java

               

              where can be seen

               

              CoderResult result = charsetEncoder.encode(buffer, out, true);

                                  if (result.isOverflow()) {

                                      outputStream.flushInternal();

                                      if (out.remaining() == 0) {

                                          outputStream.close();

                                          error = true;

                                          return;

                                      }

                                  } else {

                                      done = true;

                                  }

               

              which imo is a bug, because as it seems

              this subclass of java.nio.charset.CharsetEncoder.encode

              invoking sun.nio.cs.UTF_8$Encoder.encodeLoop always returns  overflow and the situation when out.remaining() !=0  can lead to neither 'error', neither 'done' and the cycle is endless

               

              any idea how we should proceed from here ?

              • 4. Re: worker tasks hangs on connection reset on wildfly 10.1.0.Final
                jaikiran pai Master

                This is something that @ctomc or swd847 might know more about.

                • 5. Re: worker tasks hangs on connection reset on wildfly 10.1.0.Final
                  Stan Golemanov Newbie

                  i am a bit ignorant of the procedure, should i wait for them to see this topic and maybe answer or should i reach them by other means ?

                   

                   

                   

                  thanks

                  • 6. Re: worker tasks hangs on connection reset on wildfly 10.1.0.Final
                    Stan Golemanov Newbie

                    more info

                     

                     

                    after some experimenting we found a workaround which is to add ignore flush to the servlet settings

                     

                    <servlet-container name="default" ignore-flush="true">

                                    <jsp-config/>

                                    <websockets/>

                                </servlet-container>

                     

                    this imo is not an actual solution because of

                     

                    Ignore flush: Ignore flushes on the servlet output stream. In most cases these just hurt performance for no good reason.

                     

                    which can be found in the mini help in the wildfly management console

                     

                     

                    i am keeping this thread open, because the actual cause of the problem is still not found

                     

                     

                     

                    as a side question

                     

                    ServletOutputStreamImpl.bufferSize

                     

                    is there any way to edit it's value from the configuration files ?