worker tasks hangs on connection reset on wildfly 10.1.0.Final
stan_hg Oct 6, 2017 10:07 AMHi,
our setup is as follows
Linux + Java 1.8.0_77-b03 + Wildfly10.1.0 + Spring MVC 4.01 + Thymeleaf 3.0.8 ( have the same issue with Thymeleaf 3.0.0 or/and Wildfly 10.0.0 and 8.2.0 )
We run multiple web sites projects with this setup on different machines.
One of our clients, reports that occasionally their web site stops responding and needs a restart.
After some investigations, we found that all of the worker threads ends( 2 core processor, so by default 32 threads) in permanent RUNNABLE state.
This at first seemed that it happens randomly, but with some more investigations we found a way to hang a thread almost every time
it requires to use wget on some of the urls on the site and stops it ( ctrl+c) before it end naturally ( can be achieved with killing the browser too )
it looks like this
wget -r -T 60 http://192.168.10.183:8081/bg/test2/
--2017-10-06 16:08:58-- http://192.168.10.183:8081/bg/test2/
Connecting to 192.168.10.183:8081... connected.
HTTP request sent, awaiting response...
Ctrl+c here
using JProfilier it looks like Gyazo - 3ddf639bd256a18637f982086304f6b2.png as task 2 never ends RUNNABLE state
thread dumping shows that that thread locks
dump just after wget command
"default task-2" #128 prio=5 os_prio=0 tid=0x0000000002a8a800 nid=0x407a runnable [0x00007fd967ffb000]
java.lang.Thread.State: RUNNABLE
at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:150)
at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:296)
at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:300)
at io.undertow.servlet.spec.ServletPrintWriterDelegate.write(ServletPrintWriterDelegate.java:129)
at org.thymeleaf.engine.OpenElementTag.write(OpenElementTag.java:148)
at org.thymeleaf.engine.OutputTemplateHandler.handleOpenElement(OutputTemplateHandler.java:142)
at org.thymeleaf.engine.ProcessorTemplateHandler.handleOpenElement(ProcessorTemplateHandler.java:1576)
at org.thymeleaf.engine.OpenElementTag.beHandled(OpenElementTag.java:205)
at org.thymeleaf.engine.Model.process(Model.java:282)
at org.thymeleaf.engine.Model.process(Model.java:290)
at org.thymeleaf.engine.IteratedGatheringModelProcessable.processIterationModel(IteratedGatheringModelProcessable.java:367)
at org.thymeleaf.engine.IteratedGatheringModelProcessable.process(IteratedGatheringModelProcessable.java:221)
at org.thymeleaf.engine.ProcessorTemplateHandler.handleCloseElement(ProcessorTemplateHandler.java:1640)
at org.thymeleaf.engine.CloseElementTag.beHandled(CloseElementTag.java:139)
at org.thymeleaf.engine.Model.process(Model.java:282)
at org.thymeleaf.engine.ProcessorTemplateHandler.handleOpenElement(ProcessorTemplateHandler.java:1587)
at org.thymeleaf.engine.OpenElementTag.beHandled(OpenElementTag.java:205)
at org.thymeleaf.engine.Model.process(Model.java:282)
at org.thymeleaf.engine.Model.process(Model.java:290)
at org.thymeleaf.engine.GatheringModelProcessable.process(GatheringModelProcessable.java:78)
at org.thymeleaf.engine.ProcessorTemplateHandler.handleStandaloneElement(ProcessorTemplateHandler.java:993)
at org.thymeleaf.engine.StandaloneElementTag.beHandled(StandaloneElementTag.java:228)
at org.thymeleaf.engine.Model.process(Model.java:282)
at org.thymeleaf.engine.Model.process(Model.java:290)
at org.thymeleaf.engine.IteratedGatheringModelProcessable.processIterationModel(IteratedGatheringModelProcessable.java:367)
at org.thymeleaf.engine.IteratedGatheringModelProcessable.process(IteratedGatheringModelProcessable.java:221)
at org.thymeleaf.engine.ProcessorTemplateHandler.handleCloseElement(ProcessorTemplateHandler.java:1640)
at org.thymeleaf.engine.CloseElementTag.beHandled(CloseElementTag.java:139)
at org.thymeleaf.engine.TemplateModel.process(TemplateModel.java:136)
at org.thymeleaf.engine.TemplateManager.parseAndProcess(TemplateManager.java:592)
at org.thymeleaf.TemplateEngine.process(TemplateEngine.java:1098)
at org.thymeleaf.TemplateEngine.process(TemplateEngine.java:1072)
at org.thymeleaf.spring4.view.ThymeleafView.renderFragment(ThymeleafView.java:353)
at org.thymeleaf.spring4.view.ThymeleafView.render(ThymeleafView.java:191)
at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1225)
at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1012)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at unisoftbg.web.app.AppDispatchServlet.doDispatch(AppDispatchServlet.java:63)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:876)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at unisoftbg.web.filter.HeaderFilter.doFilter(HeaderFilter.java:91)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000006ef6c3150> (a java.util.concurrent.ThreadPoolExecutor$Worker)
dump after some time
"default task-2" #128 prio=5 os_prio=0 tid=0x0000000002a8a800 nid=0x407a runnable [0x00007fd967ffe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.cs.UTF_8$Encoder.encodeBufferLoop(UTF_8.java:645)
at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:693)
at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:579)
at io.undertow.servlet.spec.ServletPrintWriter.close(ServletPrintWriter.java:104)
at io.undertow.servlet.spec.ServletPrintWriterDelegate.close(ServletPrintWriterDelegate.java:99)
at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:470)
at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:560)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:331)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000006ef6c3150> (a java.util.concurrent.ThreadPoolExecutor$Worker)
It's worth mentioning that there are no exceptions in the server.log or in the console.
Also with wget we manage to reproduce the problem only when we reset a connection to url which serves Thymeleaf template file with th:include in it.
We realize that this may be a thymeleaf issue but we can't pinpoint the exact problem.
Any directions for solving the problem will be much appreciated.
Thanks