0 Replies Latest reply on May 22, 2018 1:46 AM by Punith Kumar

    High CPU usage due to exceptional stream close scenario

    Punith Kumar Newbie

      Platform :

      Linux linux-CMkUuJ 3.0.101-0.47.105-default #1 SMP Thu Jun 22 17:13:58 UTC 2017 (e1c7e3b) x86_64 x86_64 x86_64 GNU/Linux

       

      Lib versions:

      ./resteasy-client-3.0.16.Final.jar

      ./resteasy-jdk-http-3.0.16.Final.jar

      ./resteasy-jaxrs-3.0.16.Final.jar

      ./resteasy-jackson-provider-3.0.16.Final.jar

      ./jboss-jaxrs-api_2.0_spec-1.0.0.Final.jar

      ./jboss-logging-3.1.4.GA.jar

      ./jboss-annotations-api_1.2_spec-1.0.0.Final.jar

      ./jackson-xc-1.9.12.jar

      ./jackson-jaxrs-1.9.12.jar

      ./jackson-core-asl-1.9.2.jar

      ./jackson-mapper-asl-1.9.2.jar

      ./jaxrs-api-3.0.9.Final.jar

       

      JStack : Lot of threads in RUNNABLE state, waiting to close the stream. Consuming most of the CPU.

      "pool-8-thread-24803" #227238 prio=5 os_prio=0 tid=0x00007f1e0484f000 nid=0x3019 runnable [0x00007f1deb3f1000]

         java.lang.Thread.State: RUNNABLE

      at sun.net.httpserver.LeftOverInputStream.drain(LeftOverInputStream.java:119)

      at sun.net.httpserver.LeftOverInputStream.close(LeftOverInputStream.java:69)

      at sun.net.httpserver.FixedLengthOutputStream.close(FixedLengthOutputStream.java:96)

      at sun.net.httpserver.PlaceholderOutputStream.close(ExchangeImpl.java:454)

      at org.jboss.resteasy.plugins.server.sun.http.ResteasyHttpHandler.handle(ResteasyHttpHandler.java:85)

      at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)

      at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)

      at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)

      at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)

      at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)

      at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:645)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

      Application logs show:

      An exception during read operation, this caused the thread to hang.

       

      2018-05-12 04:17:55,329 ERROR [pool-8-thread-24803][ROOT][org.jboss.resteasy.core.ExceptionHandler 150] RESTEASY002005: Failed executing POST XXXXXXXXXXXXXXXXX

      org.jboss.resteasy.spi.ReaderException: java.io.IOException: InputStream.read() returned 0 characters when trying to read 4000 bytes

      at org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:184)

      at org.jboss.resteasy.core.MethodInjectorImpl.injectArguments(MethodInjectorImpl.java:91)

      at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:114)

      at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295)

      at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249)

      at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:236)

      at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:395)

      at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)

      at org.jboss.resteasy.plugins.server.sun.http.ResteasyHttpHandler.handle(ResteasyHttpHandler.java:68)

      at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)

      at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)

      at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)

      at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)

      at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)

      at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:645)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

      Caused by: java.io.IOException: InputStream.read() returned 0 characters when trying to read 4000 bytes

      at org.codehaus.jackson.impl.Utf8StreamParser.loadMore(Utf8StreamParser.java:182)

      at org.codehaus.jackson.impl.Utf8StreamParser._skipWSOrEnd(Utf8StreamParser.java:2304)

      at org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:438)

      at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2761)

      at org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2682)

      at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1308)

      at org.codehaus.jackson.jaxrs.JacksonJsonProvider.readFrom(JacksonJsonProvider.java:419)

      at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.readFrom(AbstractReaderInterceptorContext.java:61)

      at org.jboss.resteasy.core.interception.ServerReaderInterceptorContext.readFrom(ServerReaderInterceptorContext.java:60)

      at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:53)

      at org.jboss.resteasy.plugins.interceptors.encoding.GZIPDecodingInterceptor.aroundReadFrom(GZIPDecodingInterceptor.java:59)

      at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:55)

      at org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:151)

      ... 17 more

       

      2018-05-12 04:17:55,475 ERROR [pool-8-thread-24803][ROOT][org.jboss.resteasy.plugins.server.sun.http.ResteasyHttpHandler 76] RESTEASY017515: WTF!

      org.jboss.resteasy.spi.UnhandledException: java.io.IOException: output stream is closed

      at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:180)

      at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:411)

      at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)

      at org.jboss.resteasy.plugins.server.sun.http.ResteasyHttpHandler.handle(ResteasyHttpHandler.java:68)

      at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)

      at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)

      at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)

      at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)

      at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)

      at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:645)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

      Caused by: java.io.IOException: output stream is closed

      at sun.net.httpserver.SSLStreams$OutputStream.write(SSLStreams.java:624)

      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)

      at sun.net.httpserver.ExchangeImpl.sendResponseHeaders(ExchangeImpl.java:261)

      at sun.net.httpserver.HttpsExchangeImpl.sendResponseHeaders(HttpsExchangeImpl.java:86)

      at org.jboss.resteasy.plugins.server.sun.http.HttpServerResponse.commitHeaders(HttpServerResponse.java:107)

      at org.jboss.resteasy.plugins.server.sun.http.HttpServerResponse$1.write(HttpServerResponse.java:47)

      at org.jboss.resteasy.util.CommitHeaderOutputStream.write(CommitHeaderOutputStream.java:64)

      at org.jboss.resteasy.plugins.providers.StringTextStar.writeTo(StringTextStar.java:63)

      at org.jboss.resteasy.plugins.providers.StringTextStar.writeTo(StringTextStar.java:22)

      at org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.writeTo(AbstractWriterInterceptorContext.java:131)

      at org.jboss.resteasy.core.interception.ServerWriterInterceptorContext.writeTo(ServerWriterInterceptorContext.java:60)

      at org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.proceed(AbstractWriterInterceptorContext.java:120)

      at org.jboss.resteasy.plugins.interceptors.encoding.GZIPEncodingInterceptor.aroundWriteTo(GZIPEncodingInterceptor.java:100)

      at org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.proceed(AbstractWriterInterceptorContext.java:124)

      at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:98)

      at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:176)

      ... 12 more

       

       

       

      As seen from the exception trace and jstack trace, we find read inputstream caused abnormal behavior.

      I googled about this problem, but can't find any pointers related to this.

       

      Any references to this problem, how can application ensure the thread exits in spite of exception ?