-
15. Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Jan 12, 2015 5:22 AM (in response to ctomc)Cool, thanks. I'll install a parallell 8.2 installation and do the switch next time a boot becomes necessary. The results should be evident (or not) within a few hours...
-
16. Re: Wildfly 8.1.0.Final grinding to halt (panic)
swd847 Jan 12, 2015 3:38 PM (in response to nickarls)I don't know if 8.2 will help. I have done up an XNIO SSL patch that should help if my theory is correct:
-
17. Re: Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Jan 13, 2015 2:42 AM (in response to swd847)Thanks for the patch. I tried building it but get a test failure at
Exception in thread "Thread-866" java.lang.AssertionError: expected:<96> but was:<0> at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.failNotEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:126) at org.junit.Assert.assertEquals(Assert.java:470) at org.junit.Assert.assertEquals(Assert.java:454) at org.xnio.ssl.ConnectedSslStreamChannelReadWriteTestCase$WriteRunnable.run(ConnectedSslStreamChannelReadWriteTestCase.java:524) at java.lang.Thread.run(Thread.java:745)
------------------------------------------------------------------------------- Test set: org.xnio.ssl.ConnectedSslStreamChannelReadWriteTestCase ------------------------------------------------------------------------------- Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 500.358 sec <<< FAILURE! multipleFeedReadAndWriteWithSimpleHandshake(org.xnio.ssl.ConnectedSslStreamChannelReadWriteTestCase) Time elapsed: 500.078 sec <<< FAILURE! org.junit.ComparisonFailure: expected total size: 94 actual length: 32 expected:<{handshake data}{m[ore data}{data}{more data}{data}{more data}{more data}{data}{m]essage closed}> but was:<{handshake data}{m[]essage closed}> at org.junit.Assert.assertEquals(Assert.java:123) at org.xnio.AssertReadWrite.assertWrittenMessage(AssertReadWrite.java:91) at org.xnio.ssl.AbstractConnectedSslStreamChannelTest.assertWrittenMessage(AbstractConnectedSslStreamChannelTest.java:65) at org.xnio.ssl.ConnectedSslStreamChannelReadWriteTestCase.multipleFeedReadAndWriteWithSimpleHandshake(ConnectedSslStreamChannelReadWriteTestCase.java:220) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:66) at org.jmock.integration.junit4.JMock$1.invoke(JMock.java:37) at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:105) at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:86) at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:94) at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:84) at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:49) at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:98) at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:61) at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:54) at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34) at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44) at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:52) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:119) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103) at com.sun.proxy.$Proxy0.invoke(Unknown Source) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:150) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:91) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69)
Can the tests be run with a normal "clean install" or does it need any special profile? I assume the test were run when the pull request was merged so I'm doing something wrong?
To be specific, I tried applying the patch to the XNIO 3.2.0.Final, is it no go? Or can I use the 3.x and does that one work with WildFly 8.1.0.Final?
-
18. Re: Re: Wildfly 8.1.0.Final grinding to halt (panic)
swd847 Jan 13, 2015 3:18 AM (in response to nickarls)Just run with -Dmaven.test.skip=true, the XNIO tests are not 100% reliable on all platforms.
-
19. Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Jan 13, 2015 10:40 AM (in response to swd847)*facepalm*
It's starting to look that it wasn't a straight-and-simple WildFly issue
I upgraded to WildFly 8.2.0 and patched a 3.3.0 Final of the XNIO API for it but the issue remained. Then I noticed that the server (Linux) run a iptables prerouting forward rule for 443 -> 8443 where the server was listening at. I removed the firewall rules and ran the server as root on 443 directly and it doesn't appear (fingers crossed again) to collect hanging connections. The same setup was no problem for 7.1.1 but something(tm) has changed apparently...
Hope this helps someone else stumbling on this, thanks for your time and sorry I wasted it ;-)
-
20. Re: Wildfly 8.1.0.Final grinding to halt (panic)
swd847 Jan 13, 2015 1:27 PM (in response to nickarls)That is odd, it still sounds like some kind of Wildfly issue though.
-
21. Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Jan 13, 2015 3:05 PM (in response to swd847)Back to the drawing board. The connections still accumulate even when run without iptables port forward, WildFly 8.2.0 with the patched 3.3.0 XNIO :-/
Trying downgrading the running JRE from 8 to 7...
-
22. Re: Wildfly 8.1.0.Final grinding to halt (panic)
swd847 Jan 13, 2015 8:13 PM (in response to nickarls)Try this patch: Comparing xnio:3.x...stuartwdouglas:ssl-leak · stuartwdouglas/xnio · GitHub I think there may have still been some situations where the last one might not be 100% correct.
-
23. Re: Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Jan 14, 2015 2:31 AM (in response to swd847)Thanks for the patch, the connections still accumulate, though. Here is the latest stacktrace
2015-01-14 09:22:14,041 WARNING [javax.enterprise.resource.webcontainer.jsf.application] (default task-11) : java.io.IOException: Vastapää sulki yhteyden at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [rt.jar:1.8.0_25] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) [rt.jar:1.8.0_25] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [rt.jar:1.8.0_25] at sun.nio.ch.IOUtil.write(IOUtil.java:51) [rt.jar:1.8.0_25] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470) [rt.jar:1.8.0_25] at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:152) at org.xnio.conduits.AbstractStreamSinkConduit.write(AbstractStreamSinkConduit.java:51) at org.xnio.ssl.JsseSslStreamSinkConduit.writeWrappedBuffer(JsseSslStreamSinkConduit.java:213) at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:115) at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:77) at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:244) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:596) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.conduits.ChunkedStreamSinkConduit.doWrite(ChunkedStreamSinkConduit.java:162) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.conduits.ChunkedStreamSinkConduit.write(ChunkedStreamSinkConduit.java:126) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150) at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:239) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:1850) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:567) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:220) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at java.nio.channels.Channels$WritableByteChannelImpl.write(Channels.java:458) [rt.jar:1.8.0_25] at com.sun.faces.application.resource.ResourceHandlerImpl.handleResourceRequest(ResourceHandlerImpl.java:343) [jsf-impl-2.2.6-jbossorg-4.jar:] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at fi.affecto.marela.framework.jsf.resourcehandler.ResourceHandler.handleResourceRequest(ResourceHandler.java:66) [framework-1.0.1.jar:] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at org.icefaces.impl.push.servlet.ICEpushResourceHandler.handleResourceRequest(ICEpushResourceHandler.java:128) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.push.DynamicResourceDispatcher.handleResourceRequest(DynamicResourceDispatcher.java:78) [icefaces-4.0.0-RC1.jar:] at org.icefaces.application.ResourceRegistry.handleSessionAwareResourceRequest(ResourceRegistry.java:158) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.application.WindowScopeManager.handleSessionAwareResourceRequest(WindowScopeManager.java:72) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at org.icefaces.impl.application.ClientDescriptorSetup.handleSessionAwareResourceRequest(ClientDescriptorSetup.java:44) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.application.SessionTimeoutMonitor.handleSessionAwareResourceRequest(SessionTimeoutMonitor.java:63) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at org.icefaces.ace.component.qrcode.generator.QRCodeResourceHandler.handleResourceRequest(QRCodeResourceHandler.java:67) [icefaces-ace-4.0.0-RC1.jar:] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at fi.affecto.marela.framework.jsf.resourcehandler.ResourceHandler.handleResourceRequest(ResourceHandler.java:66) [framework-1.0.1.jar:] at org.icefaces.impl.application.AuxUploadResourceHandler.handleResourceRequest(AuxUploadResourceHandler.java:82) [icefaces-4.0.0-RC1.jar:] at javax.faces.webapp.FacesServlet.service(FacesServlet.java:643) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at fi.affecto.marela.framework.logging.MDCFilter.doFilter(MDCFilter.java:33) [framework-1.0.1.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at org.apache.deltaspike.servlet.impl.event.EventBridgeFilter.doFilter(EventBridgeFilter.java:59) [deltaspike-servlet-module-impl-1.0.2.jar:1.0.2] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at org.apache.deltaspike.servlet.impl.produce.RequestResponseHolderFilter.doFilter(RequestResponseHolderFilter.java:63) [deltaspike-servlet-module-impl-1.0.2.jar:1.0.2] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at fi.affecto.marela.framework.audit.SessionTimedOutFilter.doFilter(SessionTimedOutFilter.java:55) [framework-1.0.1.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:61) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:56) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:45) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:63) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:261) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:247) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:76) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:166) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:759) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_25] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_25] 2015-01-14 09:22:14,044 ERROR [io.undertow.request] (default task-11) UT005023: Exception handling request to /OSTi/faces/javax.faces.resource/jquery-1.11.0.min.js: java.nio.BufferOverflowException at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363) [rt.jar:1.8.0_25] at java.nio.ByteBuffer.put(ByteBuffer.java:859) [rt.jar:1.8.0_25] at io.undertow.util.HttpString.appendTo(HttpString.java:204) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:166) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:166) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:596) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.conduits.ChunkedStreamSinkConduit.doWrite(ChunkedStreamSinkConduit.java:168) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.conduits.ChunkedStreamSinkConduit.writeFinal(ChunkedStreamSinkConduit.java:230) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at org.xnio.conduits.ConduitStreamSinkChannel.writeFinal(ConduitStreamSinkChannel.java:104) at io.undertow.channels.DetachableStreamSinkChannel.writeFinal(DetachableStreamSinkChannel.java:194) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.HttpServerExchange$WriteDispatchChannel.writeFinal(HttpServerExchange.java:1829) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:565) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:600) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at java.nio.channels.Channels$WritableByteChannelImpl.implCloseChannel(Channels.java:469) [rt.jar:1.8.0_25] at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115) [rt.jar:1.8.0_25] at com.sun.faces.application.resource.ResourceHandlerImpl.handleResourceRequest(ResourceHandlerImpl.java:358) [jsf-impl-2.2.6-jbossorg-4.jar:] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at fi.affecto.marela.framework.jsf.resourcehandler.ResourceHandler.handleResourceRequest(ResourceHandler.java:66) [framework-1.0.1.jar:] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at org.icefaces.impl.push.servlet.ICEpushResourceHandler.handleResourceRequest(ICEpushResourceHandler.java:128) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.push.DynamicResourceDispatcher.handleResourceRequest(DynamicResourceDispatcher.java:78) [icefaces-4.0.0-RC1.jar:] at org.icefaces.application.ResourceRegistry.handleSessionAwareResourceRequest(ResourceRegistry.java:158) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.application.WindowScopeManager.handleSessionAwareResourceRequest(WindowScopeManager.java:72) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at org.icefaces.impl.application.ClientDescriptorSetup.handleSessionAwareResourceRequest(ClientDescriptorSetup.java:44) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.application.SessionTimeoutMonitor.handleSessionAwareResourceRequest(SessionTimeoutMonitor.java:63) [icefaces-4.0.0-RC1.jar:] at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:596) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at org.icefaces.ace.component.qrcode.generator.QRCodeResourceHandler.handleResourceRequest(QRCodeResourceHandler.java:67) [icefaces-ace-4.0.0-RC1.jar:] at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at fi.affecto.marela.framework.jsf.resourcehandler.ResourceHandler.handleResourceRequest(ResourceHandler.java:66) [framework-1.0.1.jar:] at org.icefaces.impl.application.AuxUploadResourceHandler.handleResourceRequest(AuxUploadResourceHandler.java:82) [icefaces-4.0.0-RC1.jar:] at javax.faces.webapp.FacesServlet.service(FacesServlet.java:643) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at fi.affecto.marela.framework.logging.MDCFilter.doFilter(MDCFilter.java:33) [framework-1.0.1.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at org.apache.deltaspike.servlet.impl.event.EventBridgeFilter.doFilter(EventBridgeFilter.java:59) [deltaspike-servlet-module-impl-1.0.2.jar:1.0.2] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at org.apache.deltaspike.servlet.impl.produce.RequestResponseHolderFilter.doFilter(RequestResponseHolderFilter.java:63) [deltaspike-servlet-module-impl-1.0.2.jar:1.0.2] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at fi.affecto.marela.framework.audit.SessionTimedOutFilter.doFilter(SessionTimedOutFilter.java:55) [framework-1.0.1.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:61) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:56) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:45) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:63) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.conduits.ChunkedStreamSinkConduit.doWrite(ChunkedStreamSinkConduit.java:168) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:261) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:247) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:76) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:166) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:759) [undertow-core-1.1.0.Final.jar:1.1.0.Final] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_25] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_25]
-
24. Re: Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Feb 3, 2015 5:13 AM (in response to nickarls)After all the patches on the 8.2.0 the leak is not as bad as before but it's still there. We have a script that reboots the server during the night when the usage is light so the users don't notice it that much but it's still worrying.
We have also seen the issue on two other customers (not that bad, though. These are win64 machines) but my bosses are lining up behind my desk :-/ Is anyone else experiencing memory leaks with SSL connections? One would think it would be practically impossible to run an OOTB 8.1.0 with SSL enabled without running into issues. Any experiences to share?
-
25. Re: Wildfly 8.1.0.Final grinding to halt (panic)
nickarls Feb 5, 2015 2:09 AM (in response to nickarls)Today I'll try upstream master in production, it's not like the situation can degrade (much) from the status quo. Hope your test suite is really good ;-)
-
26. Re: Wildfly 8.1.0.Final grinding to halt (panic)
carlos-rodriguez-aguado Apr 30, 2015 7:01 AM (in response to nickarls)Hi Nicklas,
Did you manage to find a solution for this?
We are having a similar situation. We get this error constantly, and in several days our server crashes out of memory. I have checked lots of sockets keep opened too.
09:01:10,690 ERROR [io.undertow.request] (default task-206) Blocking request failed HttpServerExchange{ GET /modulab/intralab/images/calendar.png}: java.lang.RuntimeException: java.io.IOException: Connessione in corso interrotta forzatamente dall'host remoto
at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:527)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:287)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_20]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_20]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_20]
Caused by: java.io.IOException: Connessione in corso interrotta forzatamente dall'host remoto
at sun.nio.ch.SocketDispatcher.write0(Native Method) [rt.jar:1.8.0_20]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51) [rt.jar:1.8.0_20]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [rt.jar:1.8.0_20]
at sun.nio.ch.IOUtil.write(IOUtil.java:51) [rt.jar:1.8.0_20]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470) [rt.jar:1.8.0_20]
at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:150) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.conduits.AbstractStreamSinkConduit.write(AbstractStreamSinkConduit.java:51) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ssl.JsseSslStreamSinkConduit.writeWrappedBuffer(JsseSslStreamSinkConduit.java:213) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:100) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:72) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:212)
at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)
at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:205)
at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)
at org.xnio.channels.Channels.flushBlocking(Channels.java:63) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:625)
at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:451)
at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:525)
... 9 more
09:01:10,690 ERROR [stderr] (default task-206) Exception in thread "default task-206" java.nio.BufferOverflowException
09:01:10,690 ERROR [stderr] (default task-206) at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363)
09:01:10,690 ERROR [stderr] (default task-206) at java.nio.ByteBuffer.put(ByteBuffer.java:859)
09:01:10,690 ERROR [stderr] (default task-206) at io.undertow.util.HttpString.appendTo(HttpString.java:204)
09:01:10,690 ERROR [stderr] (default task-206) at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:150)
09:01:10,690 ERROR [stderr] (default task-206) at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)
09:01:10,690 ERROR [stderr] (default task-206) at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:205)
09:01:10,690 ERROR [stderr] (default task-206) at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
09:01:10,705 ERROR [stderr] (default task-206) at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)
09:01:10,705 ERROR [stderr] (default task-206) at io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1489)
09:01:10,705 ERROR [stderr] (default task-206) at io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1470)
09:01:10,705 ERROR [stderr] (default task-206) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:201)
09:01:10,705 ERROR [stderr] (default task-206) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)
09:01:10,705 ERROR [stderr] (default task-206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
09:01:10,721 ERROR [stderr] (default task-206) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Thanks a lot!
Carlos