14 Replies Latest reply on Jul 28, 2017 1:18 AM by mnovak

    OutOfMemory (Stack Heap) Exception while running Wildfly 8.1.0 Final

    patrick.briand

      While running in a production environment, every 3-4 days Wildfly will start spitting out OutOfMemory exceptions, which render our Spring-MVC based site unusable.  Restarting Wildfly will bring the site back up, but we cannot accept this as a solution to this problem.

       

      Here are some basic pieces of information on the setup:

      • The site is accessed via a secure connection.
      • The service is sitting behind a reverse proxy.
      • JDK 1.7.0_67
      • OS: Red Hat Enterprise Linux Server release 5.8
      • Java Options: -Xms512m -Xmx2048m -XX:MaxPermSize=256m

       

      Here is a sample exception from our log file:

       

      2014-10-15 10:10:21,629 ERROR [org.xnio.listener] (default I/O-1) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError

              at sun.misc.Unsafe.allocateMemory(Native Method) [rt.jar:1.7.0_67]

              at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127) [rt.jar:1.7.0_67]

              at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) [rt.jar:1.7.0_67]

              at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.ssl.JsseSslConduitEngine.<init>(JsseSslConduitEngine.java:143) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.ssl.JsseSslStreamConnection.<init>(JsseSslStreamConnection.java:71) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:45) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:37) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.ssl.AbstractAcceptingSslChannel.accept(AbstractAcceptingSslChannel.java:187) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:289) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.nio.NioTcpServerHandle.handleReady(NioTcpServerHandle.java:53) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]

              at org.xnio.nio.WorkerThread.run(WorkerThread.java:539) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]

       

       

      Google searches for "XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError" on this revealed one post in which there were leaking sockets when being accessed via a secure connection.  This was supposedly resolved with Wildfly 8.1.0 Final.  After checking the open sockets, there does indeed appear to be an issue there, however I am unsure if this is the actual root cause of this error.  Running the following command reveals 55 socket leaks.  This is double the number we were seeing yesterday.

       

      lsof -c java | grep "identify protocol"

      java    13401  dev  438u  sock        0,5      0t0 161594234 can't identify protocol

      java    13401  dev  446u  sock        0,5      0t0 162184626 can't identify protocol

      java    13401  dev  535u  sock        0,5      0t0 161339815 can't identify protocol

      java    13401  dev  557u  sock        0,5      0t0 161537022 can't identify protocol

      java    13401  dev  566u  sock        0,5      0t0 161495973 can't identify protocol

      java    13401  dev  586u  sock        0,5      0t0 161537023 can't identify protocol

      java    13401  dev  597u  sock        0,5      0t0 161498808 can't identify protocol

      java    13401  dev  607u  sock        0,5      0t0 161302472 can't identify protocol

      java    13401  dev  619u  sock        0,5      0t0 161355706 can't identify protocol

      java    13401  dev  621u  sock        0,5      0t0 161355708 can't identify protocol

      java    13401  dev  645u  sock        0,5      0t0 161537026 can't identify protocol

      java    13401  dev  650u  sock        0,5      0t0 161449056 can't identify protocol

      java    13401  dev  686u  sock        0,5      0t0 161767080 can't identify protocol

      java    13401  dev  687u  sock        0,5      0t0 161767081 can't identify protocol

      java    13401  dev  692u  sock        0,5      0t0 161748198 can't identify protocol

      java    13401  dev  707u  sock        0,5      0t0 161706718 can't identify protocol

      java    13401  dev  712u  sock        0,5      0t0 161806925 can't identify protocol

      java    13401  dev  713u  sock        0,5      0t0 161764540 can't identify protocol

      java    13401  dev  716u  sock        0,5      0t0 161768605 can't identify protocol

      java    13401  dev  717u  sock        0,5      0t0 162185023 can't identify protocol

      java    13401  dev  726u  sock        0,5      0t0 162179060 can't identify protocol

      java    13401  dev  733u  sock        0,5      0t0 161793109 can't identify protocol

      java    13401  dev  734u  sock        0,5      0t0 162177351 can't identify protocol

      java    13401  dev  735u  sock        0,5      0t0 161801726 can't identify protocol

      java    13401  dev  765u  sock        0,5      0t0 161769466 can't identify protocol

      java    13401  dev  768u  sock        0,5      0t0 161760997 can't identify protocol

      java    13401  dev  777u  sock        0,5      0t0 162172544 can't identify protocol

      java    13401  dev  779u  sock        0,5      0t0 161804449 can't identify protocol

      java    13401  dev  780u  sock        0,5      0t0 161790944 can't identify protocol

      java    13401  dev  781u  sock        0,5      0t0 161776945 can't identify protocol

      java    13401  dev  784u  sock        0,5      0t0 161763790 can't identify protocol

      java    13401  dev  786u  sock        0,5      0t0 161810235 can't identify protocol

      java    13401  dev  791u  sock        0,5      0t0 161810345 can't identify protocol

      java    13401  dev  811u  sock        0,5      0t0 161763131 can't identify protocol

      java    13401  dev  816u  sock        0,5      0t0 161763262 can't identify protocol

      java    13401  dev  817u  sock        0,5      0t0 161845066 can't identify protocol

      java    13401  dev  825u  sock        0,5      0t0 162185087 can't identify protocol

      java    13401  dev  830u  sock        0,5      0t0 161795163 can't identify protocol

      java    13401  dev  832u  sock        0,5      0t0 162185094 can't identify protocol

      java    13401  dev  851u  sock        0,5      0t0 162180039 can't identify protocol

      java    13401  dev  852u  sock        0,5      0t0 162169525 can't identify protocol

      java    13401  dev  869u  sock        0,5      0t0 162012138 can't identify protocol

      java    13401  dev  875u  sock        0,5      0t0 162187271 can't identify protocol

      java    13401  dev  883u  sock        0,5      0t0 162190203 can't identify protocol

      java    13401  dev  887u  sock        0,5      0t0 162189445 can't identify protocol

      java    13401  dev  888u  sock        0,5      0t0 162182814 can't identify protocol

      java    13401  dev  889u  sock        0,5      0t0 162184031 can't identify protocol

      java    13401  dev  900u  sock        0,5      0t0 162170158 can't identify protocol

      java    13401  dev  903u  sock        0,5      0t0 162194352 can't identify protocol

      java    13401  dev  904u  sock        0,5      0t0 162186059 can't identify protocol

      java    13401  dev  907u  sock        0,5      0t0 162186957 can't identify protocol

      java    13401  dev  908u  sock        0,5      0t0 162194903 can't identify protocol

      java    13401  dev  911u  sock        0,5      0t0 162195173 can't identify protocol

      java    13401  dev  917u  sock        0,5      0t0 162194364 can't identify protocol

      java    13401  dev  922u  sock        0,5      0t0 162196383 can't identify protocol

       

       

      Any help with this would be appreciated.

       

      Thank you,

      Patrick Briand