High CPU usage due to exceptional stream close scenario
Platform :
Linux linux-CMkUuJ 3.0.101-0.47.105-default #1 SMP Thu Jun 22 17:13:58 UTC 2017 (e1c7e3b) x86_64 x86_64 x86_64 GNU/Linux
Lib versions:
./resteasy-client-3.0.16.Final.jar
./resteasy-jdk-http-3.0.16.Final.jar
./resteasy-jaxrs-3.0.16.Final.jar
./resteasy-jackson-provider-3.0.16.Final.jar
./jboss-jaxrs-api_2.0_spec-1.0.0.Final.jar
./jboss-logging-3.1.4.GA.jar
./jboss-annotations-api_1.2_spec-1.0.0.Final.jar
./jackson-xc-1.9.12.jar
./jackson-jaxrs-1.9.12.jar
./jackson-core-asl-1.9.2.jar
./jackson-mapper-asl-1.9.2.jar
./jaxrs-api-3.0.9.Final.jar
JStack : Lot of threads in RUNNABLE state, waiting to close the stream. Consuming most of the CPU.
"pool-8-thread-24803" #227238 prio=5 os_prio=0 tid=0x00007f1e0484f000 nid=0x3019 runnable [0x00007f1deb3f1000]
java.lang.Thread.State: RUNNABLE
at sun.net.httpserver.LeftOverInputStream.drain(LeftOverInputStream.java:119)
at sun.net.httpserver.LeftOverInputStream.close(LeftOverInputStream.java:69)
at sun.net.httpserver.FixedLengthOutputStream.close(FixedLengthOutputStream.java:96)
at sun.net.httpserver.PlaceholderOutputStream.close(ExchangeImpl.java:454)
at org.jboss.resteasy.plugins.server.sun.http.ResteasyHttpHandler.handle(ResteasyHttpHandler.java:85)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)
at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:645)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Application logs show:
An exception during read operation, this caused the thread to hang.
2018-05-12 04:17:55,329 ERROR [pool-8-thread-24803][ROOT][org.jboss.resteasy.core.ExceptionHandler 150] RESTEASY002005: Failed executing POST XXXXXXXXXXXXXXXXX
org.jboss.resteasy.spi.ReaderException: java.io.IOException: InputStream.read() returned 0 characters when trying to read 4000 bytes
at org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:184)
at org.jboss.resteasy.core.MethodInjectorImpl.injectArguments(MethodInjectorImpl.java:91)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:114)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:236)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:395)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
at org.jboss.resteasy.plugins.server.sun.http.ResteasyHttpHandler.handle(ResteasyHttpHandler.java:68)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)
at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:645)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: InputStream.read() returned 0 characters when trying to read 4000 bytes
at org.codehaus.jackson.impl.Utf8StreamParser.loadMore(Utf8StreamParser.java:182)
at org.codehaus.jackson.impl.Utf8StreamParser._skipWSOrEnd(Utf8StreamParser.java:2304)
at org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:438)
at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2761)
at org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2682)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1308)
at org.codehaus.jackson.jaxrs.JacksonJsonProvider.readFrom(JacksonJsonProvider.java:419)
at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.readFrom(AbstractReaderInterceptorContext.java:61)
at org.jboss.resteasy.core.interception.ServerReaderInterceptorContext.readFrom(ServerReaderInterceptorContext.java:60)
at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:53)
at org.jboss.resteasy.plugins.interceptors.encoding.GZIPDecodingInterceptor.aroundReadFrom(GZIPDecodingInterceptor.java:59)
at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:55)
at org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:151)
... 17 more
2018-05-12 04:17:55,475 ERROR [pool-8-thread-24803][ROOT][org.jboss.resteasy.plugins.server.sun.http.ResteasyHttpHandler 76] RESTEASY017515: WTF!
org.jboss.resteasy.spi.UnhandledException: java.io.IOException: output stream is closed
at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:180)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:411)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
at org.jboss.resteasy.plugins.server.sun.http.ResteasyHttpHandler.handle(ResteasyHttpHandler.java:68)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)
at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)
at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:645)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: output stream is closed
at sun.net.httpserver.SSLStreams$OutputStream.write(SSLStreams.java:624)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at sun.net.httpserver.ExchangeImpl.sendResponseHeaders(ExchangeImpl.java:261)
at sun.net.httpserver.HttpsExchangeImpl.sendResponseHeaders(HttpsExchangeImpl.java:86)
at org.jboss.resteasy.plugins.server.sun.http.HttpServerResponse.commitHeaders(HttpServerResponse.java:107)
at org.jboss.resteasy.plugins.server.sun.http.HttpServerResponse$1.write(HttpServerResponse.java:47)
at org.jboss.resteasy.util.CommitHeaderOutputStream.write(CommitHeaderOutputStream.java:64)
at org.jboss.resteasy.plugins.providers.StringTextStar.writeTo(StringTextStar.java:63)
at org.jboss.resteasy.plugins.providers.StringTextStar.writeTo(StringTextStar.java:22)
at org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.writeTo(AbstractWriterInterceptorContext.java:131)
at org.jboss.resteasy.core.interception.ServerWriterInterceptorContext.writeTo(ServerWriterInterceptorContext.java:60)
at org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.proceed(AbstractWriterInterceptorContext.java:120)
at org.jboss.resteasy.plugins.interceptors.encoding.GZIPEncodingInterceptor.aroundWriteTo(GZIPEncodingInterceptor.java:100)
at org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.proceed(AbstractWriterInterceptorContext.java:124)
at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:98)
at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:176)
... 12 more
As seen from the exception trace and jstack trace, we find read inputstream caused abnormal behavior.
I googled about this problem, but can't find any pointers related to this.
Any references to this problem, how can application ensure the thread exits in spite of exception ?