1 2 Previous Next 26 Replies Latest reply on Apr 30, 2015 7:01 AM by carlos-rodriguez-aguado Go to original post
      • 15. Re: Wildfly 8.1.0.Final grinding to halt (panic)
        nickarls

        Cool, thanks. I'll install a parallell 8.2 installation and do the switch next time a boot becomes necessary. The results should be evident (or not) within a few hours...

        • 16. Re: Wildfly 8.1.0.Final grinding to halt (panic)
          swd847

          I don't know if 8.2 will help. I have done up an XNIO SSL patch that should help if my theory is correct:

           

          https://github.com/xnio/xnio/pull/78

          • 17. Re: Re: Wildfly 8.1.0.Final grinding to halt (panic)
            nickarls

            Thanks for the patch. I tried building it but get a test failure at

             

            Exception in thread "Thread-866" java.lang.AssertionError: expected:<96> but was:<0>
              at org.junit.Assert.fail(Assert.java:91)
              at org.junit.Assert.failNotEquals(Assert.java:645)
              at org.junit.Assert.assertEquals(Assert.java:126)
              at org.junit.Assert.assertEquals(Assert.java:470)
              at org.junit.Assert.assertEquals(Assert.java:454)
              at org.xnio.ssl.ConnectedSslStreamChannelReadWriteTestCase$WriteRunnable.run(ConnectedSslStreamChannelReadWriteTestCase.java:524)
              at java.lang.Thread.run(Thread.java:745)
            
            

             

            -------------------------------------------------------------------------------
            Test set: org.xnio.ssl.ConnectedSslStreamChannelReadWriteTestCase
            -------------------------------------------------------------------------------
            Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 500.358 sec <<< FAILURE!
            multipleFeedReadAndWriteWithSimpleHandshake(org.xnio.ssl.ConnectedSslStreamChannelReadWriteTestCase)  Time elapsed: 500.078 sec  <<< FAILURE!
            org.junit.ComparisonFailure: expected total size: 94 actual length: 32 expected:<{handshake data}{m[ore data}{data}{more data}{data}{more data}{more data}{data}{m]essage closed}> but was:<{handshake data}{m[]essage closed}>
              at org.junit.Assert.assertEquals(Assert.java:123)
              at org.xnio.AssertReadWrite.assertWrittenMessage(AssertReadWrite.java:91)
              at org.xnio.ssl.AbstractConnectedSslStreamChannelTest.assertWrittenMessage(AbstractConnectedSslStreamChannelTest.java:65)
              at org.xnio.ssl.ConnectedSslStreamChannelReadWriteTestCase.multipleFeedReadAndWriteWithSimpleHandshake(ConnectedSslStreamChannelReadWriteTestCase.java:220)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:483)
              at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:66)
              at org.jmock.integration.junit4.JMock$1.invoke(JMock.java:37)
              at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:105)
              at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:86)
              at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:94)
              at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:84)
              at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:49)
              at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:98)
              at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:61)
              at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:54)
              at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:34)
              at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:44)
              at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:52)
              at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
              at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:119)
              at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:101)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:483)
              at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103)
              at com.sun.proxy.$Proxy0.invoke(Unknown Source)
              at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:150)
              at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:91)
              at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69)
            
            
            

             

            Can the tests be run with a normal "clean install" or does it need any special profile? I assume the test were run when the pull request was merged so I'm doing something wrong?

             

            To be specific, I tried applying the patch to the XNIO 3.2.0.Final, is it no go? Or can I use the 3.x and does that one work with WildFly 8.1.0.Final?

            • 18. Re: Re: Wildfly 8.1.0.Final grinding to halt (panic)
              swd847

              Just run with -Dmaven.test.skip=true, the XNIO tests are not 100% reliable on all platforms.

              • 19. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                nickarls

                *facepalm*

                 

                It's starting to look that it wasn't a straight-and-simple WildFly issue

                 

                I upgraded to WildFly 8.2.0 and patched a 3.3.0 Final of the XNIO API for it but the issue remained. Then I noticed that the server (Linux) run a iptables prerouting forward rule for 443 -> 8443 where the server was listening at. I removed the firewall rules and ran the server as root on 443 directly and it doesn't appear (fingers crossed again) to collect hanging connections. The same setup was no problem for 7.1.1 but something(tm) has changed apparently...

                 

                Hope this helps someone else stumbling on this, thanks for your time and sorry I wasted it ;-)

                • 20. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                  swd847

                  That is odd, it still sounds like some kind of Wildfly issue though.

                  • 21. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                    nickarls

                    Back to the drawing board. The connections still accumulate even when run without iptables port forward, WildFly 8.2.0 with the patched 3.3.0 XNIO :-/

                    Trying downgrading the running JRE from 8 to 7...

                    • 22. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                      swd847

                      Try this patch: Comparing xnio:3.x...stuartwdouglas:ssl-leak · stuartwdouglas/xnio · GitHub I think there may have still been some situations where the last one might not be 100% correct.

                      • 23. Re: Re: Wildfly 8.1.0.Final grinding to halt (panic)
                        nickarls

                        Thanks for the patch, the connections still accumulate, though. Here is the latest stacktrace

                         

                        2015-01-14 09:22:14,041 WARNING [javax.enterprise.resource.webcontainer.jsf.application] (default task-11) : java.io.IOException: Vastapää sulki yhteyden
                                at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [rt.jar:1.8.0_25]
                                at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) [rt.jar:1.8.0_25]
                                at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [rt.jar:1.8.0_25]
                                at sun.nio.ch.IOUtil.write(IOUtil.java:51) [rt.jar:1.8.0_25]
                                at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470) [rt.jar:1.8.0_25]
                                at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:152)
                                at org.xnio.conduits.AbstractStreamSinkConduit.write(AbstractStreamSinkConduit.java:51)
                                at org.xnio.ssl.JsseSslStreamSinkConduit.writeWrappedBuffer(JsseSslStreamSinkConduit.java:213)
                                at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:115)
                                at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:77)
                                at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:244) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:596) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.conduits.ChunkedStreamSinkConduit.doWrite(ChunkedStreamSinkConduit.java:162) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.conduits.ChunkedStreamSinkConduit.write(ChunkedStreamSinkConduit.java:126) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150)
                                at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:239) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:1850) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:567) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:220) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at java.nio.channels.Channels$WritableByteChannelImpl.write(Channels.java:458) [rt.jar:1.8.0_25]
                                at com.sun.faces.application.resource.ResourceHandlerImpl.handleResourceRequest(ResourceHandlerImpl.java:343) [jsf-impl-2.2.6-jbossorg-4.jar:]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at fi.affecto.marela.framework.jsf.resourcehandler.ResourceHandler.handleResourceRequest(ResourceHandler.java:66) [framework-1.0.1.jar:]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at org.icefaces.impl.push.servlet.ICEpushResourceHandler.handleResourceRequest(ICEpushResourceHandler.java:128) [icefaces-4.0.0-RC1.jar:]
                                at org.icefaces.impl.push.DynamicResourceDispatcher.handleResourceRequest(DynamicResourceDispatcher.java:78) [icefaces-4.0.0-RC1.jar:]
                                at org.icefaces.application.ResourceRegistry.handleSessionAwareResourceRequest(ResourceRegistry.java:158) [icefaces-4.0.0-RC1.jar:]
                                at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:]
                                at org.icefaces.impl.application.WindowScopeManager.handleSessionAwareResourceRequest(WindowScopeManager.java:72) [icefaces-4.0.0-RC1.jar:]
                                at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at org.icefaces.impl.application.ClientDescriptorSetup.handleSessionAwareResourceRequest(ClientDescriptorSetup.java:44) [icefaces-4.0.0-RC1.jar:]
                                at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:]
                                at org.icefaces.impl.application.SessionTimeoutMonitor.handleSessionAwareResourceRequest(SessionTimeoutMonitor.java:63) [icefaces-4.0.0-RC1.jar:]
                                at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at org.icefaces.ace.component.qrcode.generator.QRCodeResourceHandler.handleResourceRequest(QRCodeResourceHandler.java:67) [icefaces-ace-4.0.0-RC1.jar:]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at fi.affecto.marela.framework.jsf.resourcehandler.ResourceHandler.handleResourceRequest(ResourceHandler.java:66) [framework-1.0.1.jar:]
                                at org.icefaces.impl.application.AuxUploadResourceHandler.handleResourceRequest(AuxUploadResourceHandler.java:82) [icefaces-4.0.0-RC1.jar:]
                                at javax.faces.webapp.FacesServlet.service(FacesServlet.java:643) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at fi.affecto.marela.framework.logging.MDCFilter.doFilter(MDCFilter.java:33) [framework-1.0.1.jar:]
                                at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at org.apache.deltaspike.servlet.impl.event.EventBridgeFilter.doFilter(EventBridgeFilter.java:59) [deltaspike-servlet-module-impl-1.0.2.jar:1.0.2]
                                at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at org.apache.deltaspike.servlet.impl.produce.RequestResponseHolderFilter.doFilter(RequestResponseHolderFilter.java:63) [deltaspike-servlet-module-impl-1.0.2.jar:1.0.2]
                                at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at fi.affecto.marela.framework.audit.SessionTimedOutFilter.doFilter(SessionTimedOutFilter.java:55) [framework-1.0.1.jar:]
                                at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:61) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:56) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:45) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:63) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:261) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:247) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:76) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:166) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:759) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_25]
                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_25]
                                at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_25]
                        
                        
                        2015-01-14 09:22:14,044 ERROR [io.undertow.request] (default task-11) UT005023: Exception handling request to /OSTi/faces/javax.faces.resource/jquery-1.11.0.min.js: java.nio.BufferOverflowException
                                at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363) [rt.jar:1.8.0_25]
                                at java.nio.ByteBuffer.put(ByteBuffer.java:859) [rt.jar:1.8.0_25]
                                at io.undertow.util.HttpString.appendTo(HttpString.java:204) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:166) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:166) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:596) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.conduits.ChunkedStreamSinkConduit.doWrite(ChunkedStreamSinkConduit.java:168) [undertow-core-1.1.0.Final.jar:1.1.0.Final]                  
                                at io.undertow.conduits.ChunkedStreamSinkConduit.writeFinal(ChunkedStreamSinkConduit.java:230) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at org.xnio.conduits.ConduitStreamSinkChannel.writeFinal(ConduitStreamSinkChannel.java:104)
                                at io.undertow.channels.DetachableStreamSinkChannel.writeFinal(DetachableStreamSinkChannel.java:194) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.HttpServerExchange$WriteDispatchChannel.writeFinal(HttpServerExchange.java:1829) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:565) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:600) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at java.nio.channels.Channels$WritableByteChannelImpl.implCloseChannel(Channels.java:469) [rt.jar:1.8.0_25]
                                at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115) [rt.jar:1.8.0_25]
                                at com.sun.faces.application.resource.ResourceHandlerImpl.handleResourceRequest(ResourceHandlerImpl.java:358) [jsf-impl-2.2.6-jbossorg-4.jar:]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at fi.affecto.marela.framework.jsf.resourcehandler.ResourceHandler.handleResourceRequest(ResourceHandler.java:66) [framework-1.0.1.jar:]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at org.icefaces.impl.push.servlet.ICEpushResourceHandler.handleResourceRequest(ICEpushResourceHandler.java:128) [icefaces-4.0.0-RC1.jar:]              
                                at org.icefaces.impl.push.DynamicResourceDispatcher.handleResourceRequest(DynamicResourceDispatcher.java:78) [icefaces-4.0.0-RC1.jar:]              
                                at org.icefaces.application.ResourceRegistry.handleSessionAwareResourceRequest(ResourceRegistry.java:158) [icefaces-4.0.0-RC1.jar:]           
                                at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:]
                                at org.icefaces.impl.application.WindowScopeManager.handleSessionAwareResourceRequest(WindowScopeManager.java:72) [icefaces-4.0.0-RC1.jar:]
                                at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at org.icefaces.impl.application.ClientDescriptorSetup.handleSessionAwareResourceRequest(ClientDescriptorSetup.java:44) [icefaces-4.0.0-RC1.jar:]
                                at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:]
                                at org.icefaces.impl.application.SessionTimeoutMonitor.handleSessionAwareResourceRequest(SessionTimeoutMonitor.java:63) [icefaces-4.0.0-RC1.jar:]
                                at org.icefaces.impl.application.SessionAwareResourceHandlerWrapper.handleResourceRequest(SessionAwareResourceHandlerWrapper.java:40) [icefaces-4.0.0-RC1.jar:]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                        
                                at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:596) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at org.icefaces.ace.component.qrcode.generator.QRCodeResourceHandler.handleResourceRequest(QRCodeResourceHandler.java:67) [icefaces-ace-4.0.0-RC1.jar:]
                                at javax.faces.application.ResourceHandlerWrapper.handleResourceRequest(ResourceHandlerWrapper.java:153) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at fi.affecto.marela.framework.jsf.resourcehandler.ResourceHandler.handleResourceRequest(ResourceHandler.java:66) [framework-1.0.1.jar:]        
                                at org.icefaces.impl.application.AuxUploadResourceHandler.handleResourceRequest(AuxUploadResourceHandler.java:82) [icefaces-4.0.0-RC1.jar:]       
                                at javax.faces.webapp.FacesServlet.service(FacesServlet.java:643) [jboss-jsf-api_2.2_spec-2.2.6.jar:2.2.6]
                                at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]       
                                at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at fi.affecto.marela.framework.logging.MDCFilter.doFilter(MDCFilter.java:33) [framework-1.0.1.jar:]                 
                                at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]    
                                at org.apache.deltaspike.servlet.impl.event.EventBridgeFilter.doFilter(EventBridgeFilter.java:59) [deltaspike-servlet-module-impl-1.0.2.jar:1.0.2]
                                at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]    
                                at org.apache.deltaspike.servlet.impl.produce.RequestResponseHolderFilter.doFilter(RequestResponseHolderFilter.java:63) [deltaspike-servlet-module-impl-1.0.2.jar:1.0.2]
                                at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]                               
                                at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]       
                                at fi.affecto.marela.framework.audit.SessionTimedOutFilter.doFilter(SessionTimedOutFilter.java:55) [framework-1.0.1.jar:]                     
                                at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:61) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)            
                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:56) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]              
                                at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:45) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:63) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.conduits.ChunkedStreamSinkConduit.doWrite(ChunkedStreamSinkConduit.java:168) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:261) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:247) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:76) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:166) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:759) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
                                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_25]
                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_25]
                                at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_25]
                        
                        • 24. Re: Re: Wildfly 8.1.0.Final grinding to halt (panic)
                          nickarls

                          After all the patches on the 8.2.0 the leak is not as bad as before but it's still there. We have a script that reboots the server during the night when the usage is light so the users don't notice it that much but it's still worrying.

                           

                          We have also seen the issue on two other customers (not that bad, though. These are win64 machines) but my bosses are lining up behind my desk :-/ Is anyone else experiencing memory leaks with SSL connections? One would think it would be practically impossible to run an OOTB 8.1.0 with SSL enabled without running into issues. Any experiences to share?

                          • 25. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                            nickarls

                            Today I'll try upstream master in production, it's not like the situation can degrade (much) from the status quo. Hope your test suite is really good ;-)

                            • 26. Re: Wildfly 8.1.0.Final grinding to halt (panic)
                              carlos-rodriguez-aguado

                              Hi Nicklas,

                               

                              Did you manage to find a solution for this?

                              We are having a similar situation. We get this error constantly, and in several days our server crashes out of memory. I have checked lots of sockets keep opened too.

                               

                              09:01:10,690 ERROR [io.undertow.request] (default task-206) Blocking request failed HttpServerExchange{ GET /modulab/intralab/images/calendar.png}: java.lang.RuntimeException: java.io.IOException: Connessione in corso interrotta forzatamente dall'host remoto

                                at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:527)

                                at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:287)

                                at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)

                                at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73)

                                at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146)

                                at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177)

                                at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)

                                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_20]

                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_20]

                                at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_20]

                              Caused by: java.io.IOException: Connessione in corso interrotta forzatamente dall'host remoto

                                at sun.nio.ch.SocketDispatcher.write0(Native Method) [rt.jar:1.8.0_20]

                                at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51) [rt.jar:1.8.0_20]

                                at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [rt.jar:1.8.0_20]

                                at sun.nio.ch.IOUtil.write(IOUtil.java:51) [rt.jar:1.8.0_20]

                                at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470) [rt.jar:1.8.0_20]

                                at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:150) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]

                                at org.xnio.conduits.AbstractStreamSinkConduit.write(AbstractStreamSinkConduit.java:51) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

                                at org.xnio.ssl.JsseSslStreamSinkConduit.writeWrappedBuffer(JsseSslStreamSinkConduit.java:213) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

                                at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:100) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

                                at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:72) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

                                at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:212)

                                at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)

                                at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:205)

                                at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

                                at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)

                                at org.xnio.channels.Channels.flushBlocking(Channels.java:63) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

                                at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:625)

                                at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:451)

                                at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:525)

                                ... 9 more

                               

                               

                              09:01:10,690 ERROR [stderr] (default task-206) Exception in thread "default task-206" java.nio.BufferOverflowException

                              09:01:10,690 ERROR [stderr] (default task-206) at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363)

                              09:01:10,690 ERROR [stderr] (default task-206) at java.nio.ByteBuffer.put(ByteBuffer.java:859)

                              09:01:10,690 ERROR [stderr] (default task-206) at io.undertow.util.HttpString.appendTo(HttpString.java:204)

                              09:01:10,690 ERROR [stderr] (default task-206) at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:150)

                              09:01:10,690 ERROR [stderr] (default task-206) at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)

                              09:01:10,690 ERROR [stderr] (default task-206) at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:205)

                              09:01:10,690 ERROR [stderr] (default task-206) at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)

                              09:01:10,705 ERROR [stderr] (default task-206) at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)

                              09:01:10,705 ERROR [stderr] (default task-206) at io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1489)

                              09:01:10,705 ERROR [stderr] (default task-206) at io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1470)

                              09:01:10,705 ERROR [stderr] (default task-206) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:201)

                              09:01:10,705 ERROR [stderr] (default task-206) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)

                              09:01:10,705 ERROR [stderr] (default task-206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                              09:01:10,721 ERROR [stderr] (default task-206) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                               

                               

                               

                              Thanks a lot!

                               

                              Carlos

                              1 2 Previous Next