worker tasks hangs on connection reset on wildfly 10.1.0.Final

[stan_hg]

Hi,

our setup is as follows

Linux + Java 1.8.0_77-b03 + Wildfly10.1.0 + Spring MVC 4.01 + Thymeleaf 3.0.8 ( have the same issue with  Thymeleaf 3.0.0 or/and Wildfly 10.0.0 and 8.2.0 )

We run multiple web sites projects with this setup on different machines.

One of our clients, reports that occasionally their web site stops responding and needs a restart.

After some investigations, we found that all of the worker threads ends( 2 core processor, so by default 32 threads) in permanent RUNNABLE state.

This at first seemed that it happens randomly, but with some more investigations we found a way to hang a thread almost every time

it requires to use wget on some of the urls on the site and stops it ( ctrl+c) before it end naturally ( can be achieved with killing the browser too )

it looks like this

wget -r -T 60  http://192.168.10.183:8081/bg/test2/

--2017-10-06 16:08:58--  http://192.168.10.183:8081/bg/test2/

Connecting to 192.168.10.183:8081... connected.

HTTP request sent, awaiting response...

Ctrl+c here

using JProfilier it looks like Gyazo - 3ddf639bd256a18637f982086304f6b2.png  as task 2 never ends RUNNABLE state

thread dumping shows that that thread locks

dump just after wget command

"default task-2" #128 prio=5 os_prio=0 tid=0x0000000002a8a800 nid=0x407a runnable [0x00007fd967ffb000]

   java.lang.Thread.State: RUNNABLE

at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:150)

at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:296)

at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:300)

at io.undertow.servlet.spec.ServletPrintWriterDelegate.write(ServletPrintWriterDelegate.java:129)

at org.thymeleaf.engine.OpenElementTag.write(OpenElementTag.java:148)

at org.thymeleaf.engine.OutputTemplateHandler.handleOpenElement(OutputTemplateHandler.java:142)

at org.thymeleaf.engine.ProcessorTemplateHandler.handleOpenElement(ProcessorTemplateHandler.java:1576)

at org.thymeleaf.engine.OpenElementTag.beHandled(OpenElementTag.java:205)

at org.thymeleaf.engine.Model.process(Model.java:282)

at org.thymeleaf.engine.Model.process(Model.java:290)

at org.thymeleaf.engine.IteratedGatheringModelProcessable.processIterationModel(IteratedGatheringModelProcessable.java:367)

at org.thymeleaf.engine.IteratedGatheringModelProcessable.process(IteratedGatheringModelProcessable.java:221)

at org.thymeleaf.engine.ProcessorTemplateHandler.handleCloseElement(ProcessorTemplateHandler.java:1640)

at org.thymeleaf.engine.CloseElementTag.beHandled(CloseElementTag.java:139)

at org.thymeleaf.engine.Model.process(Model.java:282)

at org.thymeleaf.engine.ProcessorTemplateHandler.handleOpenElement(ProcessorTemplateHandler.java:1587)

at org.thymeleaf.engine.OpenElementTag.beHandled(OpenElementTag.java:205)

at org.thymeleaf.engine.Model.process(Model.java:282)

at org.thymeleaf.engine.Model.process(Model.java:290)

at org.thymeleaf.engine.GatheringModelProcessable.process(GatheringModelProcessable.java:78)

at org.thymeleaf.engine.ProcessorTemplateHandler.handleStandaloneElement(ProcessorTemplateHandler.java:993)

at org.thymeleaf.engine.StandaloneElementTag.beHandled(StandaloneElementTag.java:228)

at org.thymeleaf.engine.Model.process(Model.java:282)

at org.thymeleaf.engine.Model.process(Model.java:290)

at org.thymeleaf.engine.IteratedGatheringModelProcessable.processIterationModel(IteratedGatheringModelProcessable.java:367)

at org.thymeleaf.engine.IteratedGatheringModelProcessable.process(IteratedGatheringModelProcessable.java:221)

at org.thymeleaf.engine.ProcessorTemplateHandler.handleCloseElement(ProcessorTemplateHandler.java:1640)

at org.thymeleaf.engine.CloseElementTag.beHandled(CloseElementTag.java:139)

at org.thymeleaf.engine.TemplateModel.process(TemplateModel.java:136)

at org.thymeleaf.engine.TemplateManager.parseAndProcess(TemplateManager.java:592)

at org.thymeleaf.TemplateEngine.process(TemplateEngine.java:1098)

at org.thymeleaf.TemplateEngine.process(TemplateEngine.java:1072)

at org.thymeleaf.spring4.view.ThymeleafView.renderFragment(ThymeleafView.java:353)

at org.thymeleaf.spring4.view.ThymeleafView.render(ThymeleafView.java:191)

at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1225)

at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1012)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)

at unisoftbg.web.app.AppDispatchServlet.doDispatch(AppDispatchServlet.java:63)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:876)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)

at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)

at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)

at unisoftbg.web.filter.HeaderFilter.doFilter(HeaderFilter.java:91)

at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)

at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)

at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)

at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)

at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

- <0x00000006ef6c3150> (a java.util.concurrent.ThreadPoolExecutor$Worker)

dump after some time

"default task-2" #128 prio=5 os_prio=0 tid=0x0000000002a8a800 nid=0x407a runnable [0x00007fd967ffe000]

   java.lang.Thread.State: RUNNABLE

at sun.nio.cs.UTF_8$Encoder.encodeBufferLoop(UTF_8.java:645)

at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:693)

at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:579)

at io.undertow.servlet.spec.ServletPrintWriter.close(ServletPrintWriter.java:104)

at io.undertow.servlet.spec.ServletPrintWriterDelegate.close(ServletPrintWriterDelegate.java:99)

at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:470)

at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:560)

at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:331)

at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

- <0x00000006ef6c3150> (a java.util.concurrent.ThreadPoolExecutor$Worker)

It's worth mentioning that there are no exceptions in the server.log or in the console.

Also with wget we manage to reproduce the problem only when we reset a connection to url which serves Thymeleaf template file with th:include in it.

We realize that this may be a thymeleaf issue but we can't pinpoint the exact problem.

Any directions for solving the problem will be much appreciated.

Thanks

[jaikiran]

Any chance you can try what's been suggested here Wildfly 10.1.0.Final threads stuck in infinite loop and see if this is related to your issue.

[stan_hg]

i saw that, but http2 is not enabled(Gyazo - 58b3d6b17752100a487b499944e619ef.png ), also the problem is the same in 8.2 which does not support http2

thanks

[stan_hg]

some more info

while thread dumping we managed to see this

sun.nio.cs.UTF_8$Encoder.overflow(java.nio.CharBuffer, int) (line: 577)

sun.nio.cs.UTF_8$Encoder.encodeBufferLoop(java.nio.CharBuffer, java.nio.ByteBuffer) (line: 654)

sun.nio.cs.UTF_8$Encoder.encodeLoop(java.nio.CharBuffer, java.nio.ByteBuffer) (line: 693)

java.nio.charset.CharsetEncoder.encode(java.nio.CharBuffer, java.nio.ByteBuffer, boolean) (line: 579)

io.undertow.servlet.spec.ServletPrintWriter.close() (line: 104)

io.undertow.servlet.spec.ServletPrintWriterDelegate.close() (line: 99)

io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter() (line: 491)

io.undertow.servlet.spec.HttpServletResponseImpl.responseDone() (line: 581)

io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(io.undertow.server.HttpServerExchange, io.undertow.servlet.handlers.ServletChain, io.undertow.servlet.handlers.ServletRequestContext, javax.servlet.ServletRequest, javax.servlet.ServletResponse) (line: 308)

io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(io.undertow.server.HttpServerExchange, io.undertow.servlet.handlers.ServletRequestContext, io.undertow.servlet.handlers.ServletChain, javax.servlet.DispatcherType) (line: 247)

io.undertow.servlet.handlers.ServletInitialHandler.access$000(io.undertow.servlet.handlers.ServletInitialHandler, io.undertow.server.HttpServerExchange, io.undertow.servlet.handlers.ServletRequestContext, io.undertow.servlet.handlers.ServletChain, javax.servlet.DispatcherType) (line: 76)

io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(io.undertow.server.HttpServerExchange) (line: 166)

io.undertow.server.Connectors.executeRootHandler(io.undertow.server.HttpHandler, io.undertow.server.HttpServerExchange) (line: 197)

io.undertow.server.HttpServerExchange$1.run() (line: 759)

java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) (line: 1142)

java.util.concurrent.ThreadPoolExecutor$Worker.run() (line: 617)

java.lang.Thread.run() (line: 745)

which lead us to

https://github.com/undertow-io/undertow/blob/master/servlet/src/main/java/io/undertow/servlet/spec/ServletPrintWriter.java

where can be seen

CoderResult result = charsetEncoder.encode(buffer, out, true);

                    if (result.isOverflow()) {

                        outputStream.flushInternal();

                        if (out.remaining() == 0) {

                            outputStream.close();

                            error = true;

                            return;

                        }

                    } else {

                        done = true;

                    }

which imo is a bug, because as it seems

this subclass of java.nio.charset.CharsetEncoder.encode

invoking sun.nio.cs.UTF_8$Encoder.encodeLoop always returns  overflow and the situation when out.remaining() !=0  can lead to neither 'error', neither 'done' and the cycle is endless

any idea how we should proceed from here ?

[jaikiran]

This is something that @ctomc or swd847 might know more about.

[stan_hg]

i am a bit ignorant of the procedure, should i wait for them to see this topic and maybe answer or should i reach them by other means ?

thanks

[stan_hg]

more info

after some experimenting we found a workaround which is to add ignore flush to the servlet settings

<servlet-container name="default" ignore-flush="true">

                <jsp-config/>

                <websockets/>

            </servlet-container>

this imo is not an actual solution because of

Ignore flush: Ignore flushes on the servlet output stream. In most cases these just hurt performance for no good reason.

which can be found in the mini help in the wildfly management console

i am keeping this thread open, because the actual cause of the problem is still not found

as a side question

ServletOutputStreamImpl.bufferSize

is there any way to edit it's value from the configuration files ?