1 2 Previous Next 26 Replies Latest reply on Apr 30, 2015 7:01 AM by Carlos Rodríguez Aguado

    Wildfly 8.1.0.Final grinding to halt (panic)

    Nicklas Karlsson Master

      We have had the problem of two our customers (one Linux, one Win64) running JDK8 having their WF 8.1.0 Final grind to a halt on OOM exceptions due to similar reasons. Both use HTTPS and the OOM comes from XNIO accepting connections. I took a heap dump of one of the servers, "top" showed 2G usage but the mem dump was not that big. Eclipse MAT showed the leak suspect

       

      1 039 instances of "sun.nio.ch.SelectionKeyImpl", loaded by "<system class loader>" occupy 39 286 040 (21,20%) bytes.

      Biggest instances:

      • sun.nio.ch.SelectionKeyImpl @ 0x32126738 - 2 157 720 (1,16%) bytes.

       

      but memory-wise it doesn't look that big(?)

       

      Thankful for suggestions!

        • 1. Re: Wildfly 8.1.0.Final grinding to halt (panic)
          Tomaz Cerar Master

          Afaik this was fixed in xnio/undertow.

          Upgrade to 8.2.0 and it should probably work.

          • 2. Re: Wildfly 8.1.0.Final grinding to halt (panic)
            Nicklas Karlsson Master

            I found this thread where it was claimed that it was fixed in 8.1.0

             

            https://developer.jboss.org/thread/237403?start=0&tstart=0

             

            Could it be something related? Does swd847 remember anything about this?

            • 3. Re: Wildfly 8.1.0.Final grinding to halt (panic)
              Stuart Douglas Master

              Is it possible that you are running out of direct buffer memory? SSL connections need 3 * 17kb buffers, and XNIO allocates these for the life of the connection. I am not sure if these will show up in your memory dump, as it is direct buffer memory rather than heap memory.

               

              Undertow 1.2 uses a new SSL implementation that only holds the buffers while they actually have content that needs to be buffered, so its SSL memory usage should be a *lot* lower.

               

              You could try increasing -XX:MaxDirectMemorySize=128Mb (or possibly something even bigger).

              1 of 1 people found this helpful
              • 4. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                Nicklas Karlsson Master

                Thanks, I'll give it a spin. I am still a bit suspicious of a leak though since the OOM didn't manifest itself during any peak hour and only on the third day after install. I also get

                 

                [user@host ~]$ /usr/sbin/lsof -p 14626 | grep "protocol" | wc -L

                192

                 

                half a day after I was forced to reboot it

                • 5. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                  jaikiran pai Master

                  Nicklas Karlsson wrote:

                   

                  Thanks, I'll give it a spin. I am still a bit suspicious of a leak though since the OOM didn't manifest itself during any peak hour and only on the third day after install.

                  Have you verified it's not this [REM3-200] Memory leak in org.xnio.ByteBufferSlicePool - JBoss Issue Tracker.  A few days back there was a similar thread here https://developer.jboss.org/message/912538#912538

                  1 of 1 people found this helpful
                  • 6. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                    Nicklas Karlsson Master

                    Thanks for the tip. I don't use JMX but I'll try adding

                     

                    -Djboss.remoting.pooled-buffers=false -XX:MaxDirectMemorySize=128M

                     

                    next time I'll have to take it down. Appears to be SSL-related mostly since I don't see those hanging connections with lsof on a sibling server that uses a normal http connector

                     

                    Can I just upgrade an 8.1.0 installation by taking the xnio/undertow jars from 8.2.0?

                    • 7. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                      Nicklas Karlsson Master

                      The number of ports in "unknown protocol" state has remained 0 after the parameters + restart so keeping my fingers crossed here...

                      • 8. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                        Nicklas Karlsson Master

                        Unfortunately, keeping fingers crossed didn't help. There are two identical servers and for some reason, it helped on one server but not on the other. The failing server still accumulates "can't identity protocol"-sockets :-/

                         

                        Thankful for suggestions

                        • 9. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                          jaikiran pai Master

                          Well, the first thing to verify is that the other server indeed has picked up the system property. Other than that I can't think of much except maybe dumping the heap (via a JVM param) on OOM errors and then analyzing it.

                          • 10. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                            Nicklas Karlsson Master

                            I've checked the console.log from both servers and it's picked up correctly in JAVA_OPTS. I'm almost certain there is some external factor involved (firewall, etc) just can't put my finger on it. I have to start looking for ways of killing off these handles or rebooting the instance during slow hours if nothing helps :-/

                            • 11. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                              Nicklas Karlsson Master

                              I run a script that once a minute counted the amount of hanging sockets and I noticed that when the count went up there was a stracktrace like

                               

                              2015-01-09 07:41:46,287 ERROR [io.undertow.request] (default task-23) Blocking request failed HttpServerExchange{ GET /MyApp/faces/javax.faces.resource/util/ace-yui.js}: java.lang.RuntimeException: java.io.IOException: Remote end has closed the connection

                                      at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:527)

                                      at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:287)

                                      at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)

                                      at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73)

                                      at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146)

                                      at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177)

                                      at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)

                                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_25]

                                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_25]

                                      at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_25]

                              Caused by: java.io.IOException:Remote end has closed the connection

                                      at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [rt.jar:1.8.0_25]

                                      at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) [rt.jar:1.8.0_25]

                                      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [rt.jar:1.8.0_25]

                                      at sun.nio.ch.IOUtil.write(IOUtil.java:51) [rt.jar:1.8.0_25]

                                      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470) [rt.jar:1.8.0_25]

                                      at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:150) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]

                                      at org.xnio.conduits.AbstractStreamSinkConduit.write(AbstractStreamSinkConduit.java:51) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

                                      at org.xnio.ssl.JsseSslStreamSinkConduit.writeWrappedBuffer(JsseSslStreamSinkConduit.java:213) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

                                      at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:100) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

                                      at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:72) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

                                      at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:212)

                                      at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)

                                      at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:205)

                                      at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

                                      at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)

                                      at org.xnio.channels.Channels.flushBlocking(Channels.java:63) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

                                      at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:625)

                                      at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:451)

                                      at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:525)

                                      ... 9 more

                               

                               

                              2015-01-09 07:41:46,293 ERROR [stderr] (default task-23) Exception in thread "default task-23" java.nio.BufferOverflowException

                              2015-01-09 07:41:46,294 ERROR [stderr] (default task-23)        at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363)

                              2015-01-09 07:41:46,294 ERROR [stderr] (default task-23)        at java.nio.ByteBuffer.put(ByteBuffer.java:859)

                              2015-01-09 07:41:46,294 ERROR [stderr] (default task-23)        at io.undertow.util.HttpString.appendTo(HttpString.java:204)

                              2015-01-09 07:41:46,294 ERROR [stderr] (default task-23)        at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:150)

                              2015-01-09 07:41:46,294 ERROR [stderr] (default task-23)        at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)

                              2015-01-09 07:41:46,294 ERROR [stderr] (default task-23)        at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:205)

                              2015-01-09 07:41:46,294 ERROR [stderr] (default task-23)        at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)

                              2015-01-09 07:41:46,295 ERROR [stderr] (default task-23)        at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)

                              2015-01-09 07:41:46,295 ERROR [stderr] (default task-23)        at io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1489)

                              2015-01-09 07:41:46,295 ERROR [stderr] (default task-23)        at io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1470)

                              2015-01-09 07:41:46,295 ERROR [stderr] (default task-23)        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:201)

                              2015-01-09 07:41:46,295 ERROR [stderr] (default task-23)        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)

                              2015-01-09 07:41:46,295 ERROR [stderr] (default task-23)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                              2015-01-09 07:41:46,295 ERROR [stderr] (default task-23)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                              2015-01-09 07:41:46,295 ERROR [stderr] (default task-23)        at java.lang.Thread.run(Thread.java:745)

                               

                              Known bug? Bad request? Something else?

                              • 12. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                                Stuart Douglas Master

                                So I think this is a leak in the XNIO SSL implementation. The underlying cause is the first exception (the second exception is a result of the first, which I should add some code to handle so that it throws an IOException instead of a buffer overflow).

                                 

                                Undertow should then call close() on the underlying connection, however it appears that there is no guarantee that XNIO will actually free the SSL buffers, as if an exception is thrown from the wrong spot they will not be closed.

                                 

                                Wildfly 9 uses a new SSL implementation, that should not have this problem.

                                • 13. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                                  Nicklas Karlsson Master

                                  So upgrading to 8.2.0 is unlikely to do any good? I'm a bit reluctant to use 9.0.0.Alpha1 in production :-/

                                  • 14. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                                    Tomaz Cerar Master

                                    Upgrading to 8.2 should definitely help. Question is to what extend.

                                    There are SSL related fixes in xnio and undertow and lots others.

                                    and when it comes to undertow / xnio stuff, 8.2.0.Final is better than 9.0.0.Alpha1 as 8.2 was tagged much later.

                                    master on github is different story.

                                    1 2 Previous Next