-
1. Re: Wildfly 8.1.0.Final grinding to halt (panic)
ctomc Dec 16, 2014 10:27 AM (in response to nickarls)Afaik this was fixed in xnio/undertow.
Upgrade to 8.2.0 and it should probably work.
-
2. Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Dec 16, 2014 1:07 PM (in response to ctomc)I found this thread where it was claimed that it was fixed in 8.1.0
https://developer.jboss.org/thread/237403?start=0&tstart=0
Could it be something related? Does swd847 remember anything about this?
-
3. Re: Wildfly 8.1.0.Final grinding to halt (panic)
swd847 Dec 16, 2014 5:47 PM (in response to nickarls)1 of 1 people found this helpfulIs it possible that you are running out of direct buffer memory? SSL connections need 3 * 17kb buffers, and XNIO allocates these for the life of the connection. I am not sure if these will show up in your memory dump, as it is direct buffer memory rather than heap memory.
Undertow 1.2 uses a new SSL implementation that only holds the buffers while they actually have content that needs to be buffered, so its SSL memory usage should be a *lot* lower.
You could try increasing
-XX:MaxDirectMemorySize=128Mb (or possibly something even bigger).
-
4. Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Dec 17, 2014 2:24 AM (in response to swd847)Thanks, I'll give it a spin. I am still a bit suspicious of a leak though since the OOM didn't manifest itself during any peak hour and only on the third day after install. I also get
[user@host ~]$ /usr/sbin/lsof -p 14626 | grep "protocol" | wc -L
192
half a day after I was forced to reboot it
-
5. Re: Wildfly 8.1.0.Final grinding to halt (panic)
jaikiran Dec 17, 2014 2:33 AM (in response to nickarls)1 of 1 people found this helpfulNicklas Karlsson wrote:
Thanks, I'll give it a spin. I am still a bit suspicious of a leak though since the OOM didn't manifest itself during any peak hour and only on the third day after install.
Have you verified it's not this [REM3-200] Memory leak in org.xnio.ByteBufferSlicePool - JBoss Issue Tracker. A few days back there was a similar thread here https://developer.jboss.org/message/912538#912538
-
6. Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Dec 17, 2014 5:25 AM (in response to jaikiran)Thanks for the tip. I don't use JMX but I'll try adding
-Djboss.remoting.pooled-buffers=false -XX:MaxDirectMemorySize=128M
next time I'll have to take it down. Appears to be SSL-related mostly since I don't see those hanging connections with lsof on a sibling server that uses a normal http connector
Can I just upgrade an 8.1.0 installation by taking the xnio/undertow jars from 8.2.0?
-
7. Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Dec 19, 2014 8:07 AM (in response to nickarls)The number of ports in "unknown protocol" state has remained 0 after the parameters + restart so keeping my fingers crossed here...
-
8. Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Jan 8, 2015 3:06 AM (in response to nickarls)Unfortunately, keeping fingers crossed didn't help. There are two identical servers and for some reason, it helped on one server but not on the other. The failing server still accumulates "can't identity protocol"-sockets :-/
Thankful for suggestions
-
9. Re: Wildfly 8.1.0.Final grinding to halt (panic)
jaikiran Jan 8, 2015 9:12 AM (in response to nickarls)Well, the first thing to verify is that the other server indeed has picked up the system property. Other than that I can't think of much except maybe dumping the heap (via a JVM param) on OOM errors and then analyzing it.
-
10. Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Jan 9, 2015 1:41 AM (in response to jaikiran)I've checked the console.log from both servers and it's picked up correctly in JAVA_OPTS. I'm almost certain there is some external factor involved (firewall, etc) just can't put my finger on it. I have to start looking for ways of killing off these handles or rebooting the instance during slow hours if nothing helps :-/
-
11. Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Jan 9, 2015 6:57 AM (in response to nickarls)I run a script that once a minute counted the amount of hanging sockets and I noticed that when the count went up there was a stracktrace like
2015-01-09 07:41:46,287 ERROR [io.undertow.request] (default task-23) Blocking request failed HttpServerExchange{ GET /MyApp/faces/javax.faces.resource/util/ace-yui.js}: java.lang.RuntimeException: java.io.IOException: Remote end has closed the connection
at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:527)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:287)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_25]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_25]
Caused by: java.io.IOException:Remote end has closed the connection
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [rt.jar:1.8.0_25]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) [rt.jar:1.8.0_25]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [rt.jar:1.8.0_25]
at sun.nio.ch.IOUtil.write(IOUtil.java:51) [rt.jar:1.8.0_25]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470) [rt.jar:1.8.0_25]
at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:150) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.conduits.AbstractStreamSinkConduit.write(AbstractStreamSinkConduit.java:51) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ssl.JsseSslStreamSinkConduit.writeWrappedBuffer(JsseSslStreamSinkConduit.java:213) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:100) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:72) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:212)
at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)
at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:205)
at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)
at org.xnio.channels.Channels.flushBlocking(Channels.java:63) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:625)
at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:451)
at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:525)
... 9 more
2015-01-09 07:41:46,293 ERROR [stderr] (default task-23) Exception in thread "default task-23" java.nio.BufferOverflowException
2015-01-09 07:41:46,294 ERROR [stderr] (default task-23) at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363)
2015-01-09 07:41:46,294 ERROR [stderr] (default task-23) at java.nio.ByteBuffer.put(ByteBuffer.java:859)
2015-01-09 07:41:46,294 ERROR [stderr] (default task-23) at io.undertow.util.HttpString.appendTo(HttpString.java:204)
2015-01-09 07:41:46,294 ERROR [stderr] (default task-23) at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:150)
2015-01-09 07:41:46,294 ERROR [stderr] (default task-23) at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)
2015-01-09 07:41:46,294 ERROR [stderr] (default task-23) at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:205)
2015-01-09 07:41:46,294 ERROR [stderr] (default task-23) at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
2015-01-09 07:41:46,295 ERROR [stderr] (default task-23) at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)
2015-01-09 07:41:46,295 ERROR [stderr] (default task-23) at io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1489)
2015-01-09 07:41:46,295 ERROR [stderr] (default task-23) at io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1470)
2015-01-09 07:41:46,295 ERROR [stderr] (default task-23) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:201)
2015-01-09 07:41:46,295 ERROR [stderr] (default task-23) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)
2015-01-09 07:41:46,295 ERROR [stderr] (default task-23) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2015-01-09 07:41:46,295 ERROR [stderr] (default task-23) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2015-01-09 07:41:46,295 ERROR [stderr] (default task-23) at java.lang.Thread.run(Thread.java:745)
Known bug? Bad request? Something else?
-
12. Re: Wildfly 8.1.0.Final grinding to halt (panic)
swd847 Jan 11, 2015 5:35 PM (in response to nickarls)So I think this is a leak in the XNIO SSL implementation. The underlying cause is the first exception (the second exception is a result of the first, which I should add some code to handle so that it throws an IOException instead of a buffer overflow).
Undertow should then call close() on the underlying connection, however it appears that there is no guarantee that XNIO will actually free the SSL buffers, as if an exception is thrown from the wrong spot they will not be closed.
Wildfly 9 uses a new SSL implementation, that should not have this problem.
-
13. Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Jan 12, 2015 1:42 AM (in response to swd847)So upgrading to 8.2.0 is unlikely to do any good? I'm a bit reluctant to use 9.0.0.Alpha1 in production :-/
-
14. Re: Wildfly 8.1.0.Final grinding to halt (panic)
ctomc Jan 12, 2015 5:16 AM (in response to nickarls)Upgrading to 8.2 should definitely help. Question is to what extend.
There are SSL related fixes in xnio and undertow and lots others.
and when it comes to undertow / xnio stuff, 8.2.0.Final is better than 9.0.0.Alpha1 as 8.2 was tagged much later.
master on github is different story.