OutOfMemory (Stack Heap) Exception while running Wildfly 8.1.0 Final
patrick.briand Oct 17, 2014 9:15 AMWhile running in a production environment, every 3-4 days Wildfly will start spitting out OutOfMemory exceptions, which render our Spring-MVC based site unusable. Restarting Wildfly will bring the site back up, but we cannot accept this as a solution to this problem.
Here are some basic pieces of information on the setup:
- The site is accessed via a secure connection.
- The service is sitting behind a reverse proxy.
- JDK 1.7.0_67
- OS: Red Hat Enterprise Linux Server release 5.8
- Java Options: -Xms512m -Xmx2048m -XX:MaxPermSize=256m
Here is a sample exception from our log file:
2014-10-15 10:10:21,629 ERROR [org.xnio.listener] (default I/O-1) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method) [rt.jar:1.7.0_67]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127) [rt.jar:1.7.0_67]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) [rt.jar:1.7.0_67]
at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ssl.JsseSslConduitEngine.<init>(JsseSslConduitEngine.java:143) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ssl.JsseSslStreamConnection.<init>(JsseSslStreamConnection.java:71) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:45) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:37) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ssl.AbstractAcceptingSslChannel.accept(AbstractAcceptingSslChannel.java:187) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:289) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.nio.NioTcpServerHandle.handleReady(NioTcpServerHandle.java:53) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.nio.WorkerThread.run(WorkerThread.java:539) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]
Google searches for "XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError" on this revealed one post in which there were leaking sockets when being accessed via a secure connection. This was supposedly resolved with Wildfly 8.1.0 Final. After checking the open sockets, there does indeed appear to be an issue there, however I am unsure if this is the actual root cause of this error. Running the following command reveals 55 socket leaks. This is double the number we were seeing yesterday.
lsof -c java | grep "identify protocol"
java 13401 dev 438u sock 0,5 0t0 161594234 can't identify protocol
java 13401 dev 446u sock 0,5 0t0 162184626 can't identify protocol
java 13401 dev 535u sock 0,5 0t0 161339815 can't identify protocol
java 13401 dev 557u sock 0,5 0t0 161537022 can't identify protocol
java 13401 dev 566u sock 0,5 0t0 161495973 can't identify protocol
java 13401 dev 586u sock 0,5 0t0 161537023 can't identify protocol
java 13401 dev 597u sock 0,5 0t0 161498808 can't identify protocol
java 13401 dev 607u sock 0,5 0t0 161302472 can't identify protocol
java 13401 dev 619u sock 0,5 0t0 161355706 can't identify protocol
java 13401 dev 621u sock 0,5 0t0 161355708 can't identify protocol
java 13401 dev 645u sock 0,5 0t0 161537026 can't identify protocol
java 13401 dev 650u sock 0,5 0t0 161449056 can't identify protocol
java 13401 dev 686u sock 0,5 0t0 161767080 can't identify protocol
java 13401 dev 687u sock 0,5 0t0 161767081 can't identify protocol
java 13401 dev 692u sock 0,5 0t0 161748198 can't identify protocol
java 13401 dev 707u sock 0,5 0t0 161706718 can't identify protocol
java 13401 dev 712u sock 0,5 0t0 161806925 can't identify protocol
java 13401 dev 713u sock 0,5 0t0 161764540 can't identify protocol
java 13401 dev 716u sock 0,5 0t0 161768605 can't identify protocol
java 13401 dev 717u sock 0,5 0t0 162185023 can't identify protocol
java 13401 dev 726u sock 0,5 0t0 162179060 can't identify protocol
java 13401 dev 733u sock 0,5 0t0 161793109 can't identify protocol
java 13401 dev 734u sock 0,5 0t0 162177351 can't identify protocol
java 13401 dev 735u sock 0,5 0t0 161801726 can't identify protocol
java 13401 dev 765u sock 0,5 0t0 161769466 can't identify protocol
java 13401 dev 768u sock 0,5 0t0 161760997 can't identify protocol
java 13401 dev 777u sock 0,5 0t0 162172544 can't identify protocol
java 13401 dev 779u sock 0,5 0t0 161804449 can't identify protocol
java 13401 dev 780u sock 0,5 0t0 161790944 can't identify protocol
java 13401 dev 781u sock 0,5 0t0 161776945 can't identify protocol
java 13401 dev 784u sock 0,5 0t0 161763790 can't identify protocol
java 13401 dev 786u sock 0,5 0t0 161810235 can't identify protocol
java 13401 dev 791u sock 0,5 0t0 161810345 can't identify protocol
java 13401 dev 811u sock 0,5 0t0 161763131 can't identify protocol
java 13401 dev 816u sock 0,5 0t0 161763262 can't identify protocol
java 13401 dev 817u sock 0,5 0t0 161845066 can't identify protocol
java 13401 dev 825u sock 0,5 0t0 162185087 can't identify protocol
java 13401 dev 830u sock 0,5 0t0 161795163 can't identify protocol
java 13401 dev 832u sock 0,5 0t0 162185094 can't identify protocol
java 13401 dev 851u sock 0,5 0t0 162180039 can't identify protocol
java 13401 dev 852u sock 0,5 0t0 162169525 can't identify protocol
java 13401 dev 869u sock 0,5 0t0 162012138 can't identify protocol
java 13401 dev 875u sock 0,5 0t0 162187271 can't identify protocol
java 13401 dev 883u sock 0,5 0t0 162190203 can't identify protocol
java 13401 dev 887u sock 0,5 0t0 162189445 can't identify protocol
java 13401 dev 888u sock 0,5 0t0 162182814 can't identify protocol
java 13401 dev 889u sock 0,5 0t0 162184031 can't identify protocol
java 13401 dev 900u sock 0,5 0t0 162170158 can't identify protocol
java 13401 dev 903u sock 0,5 0t0 162194352 can't identify protocol
java 13401 dev 904u sock 0,5 0t0 162186059 can't identify protocol
java 13401 dev 907u sock 0,5 0t0 162186957 can't identify protocol
java 13401 dev 908u sock 0,5 0t0 162194903 can't identify protocol
java 13401 dev 911u sock 0,5 0t0 162195173 can't identify protocol
java 13401 dev 917u sock 0,5 0t0 162194364 can't identify protocol
java 13401 dev 922u sock 0,5 0t0 162196383 can't identify protocol
Any help with this would be appreciated.
Thank you,
Patrick Briand