4 Replies Latest reply on Apr 24, 2015 5:31 AM by ptuomaal

    Wildfly 8.2: An existing connection was forcibly closed by the remote host

    ptuomaal

      We have upgraded from JBoss AS 6 to Wildfly 8.2. After upgrade we have noticed increase in http request errors. With JBoss AS 6 our error % was between 0.1-0.2% with an avg load or 500-700 request per minute.With wildfly it is between 0.4-0.6% (according to newrelic). However, most of the errors seems to come from this exception: java.io.IOException: An existing connection was forcibly closed by the remote host (trace visible in later message).There is an avg of 150 of those errors in 24h and that is a lot. To my knowledge this exception is thrown in cases where the server cannot write bits to the client and it should be harmeless.

       

      Have others faced the same error? Should I be worried about it and any solutions how to get rid of it?  Our app is pretty basic Java EE solution (JSP, JSF (MyFaces 2.2.), EJB3)

       

      Br,

      Petri

        • 1. Re: Wildfly 8.2: An existing connection was forcibly closed by the remote host
          jaikiran

          Can you please post the relevant portion of the log file including any exception stacktraces, here? I can't access the linked content in your original post since it asks for a login.

          • 2. Re: Wildfly 8.2: An existing connection was forcibly closed by the remote host
            ptuomaal

            Sorry about bad link, failed to make it public. Code in our app is just reading attribute from the request and it always seems to be that line causing the exception (other attributes are read earlier)

             

            code:

                private boolean isEditApplicant(HttpServletRequest request) {

                    String editApplicant = request.getParameter("editApplicant"); //this line in our code causes the exception

             

            trace:

            [io.undertow.request] (default task-38) UT005023: Exception handling request to *******.faces: java.lang.RuntimeException: java.io.IOException: An existing connection was forcibly closed by the remote host

              at io.undertow.servlet.spec.HttpServletRequestImpl.parseFormData(HttpServletRequestImpl.java:754) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.spec.HttpServletRequestImpl.getParameter(HttpServletRequestImpl.java:626) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at fi.elbit.skills.login.DefaultsFilter.isEditApplicant(DefaultsFilter.java:148) [classes:]

              at fi.elbit.skills.login.DefaultsFilter.doFilter(DefaultsFilter.java:100) [classes:]

              at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at org.apache.myfaces.webapp.filter.ExtensionsFilter.doFilter(ExtensionsFilter.java:357) [tomahawk20-1.1.14.jar:1.1.14]

              at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:61) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

              at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:56) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:45) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:63) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

              at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:261) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:247) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:76) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:166) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:759) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [rt.jar:1.8.0_31]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.8.0_31]

              at java.lang.Thread.run(Unknown Source) [rt.jar:1.8.0_31]

            Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host

              at sun.nio.ch.SocketDispatcher.read0(Native Method) [rt.jar:1.8.0_31]

              at sun.nio.ch.SocketDispatcher.read(Unknown Source) [rt.jar:1.8.0_31]

              at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) [rt.jar:1.8.0_31]

              at sun.nio.ch.IOUtil.read(Unknown Source) [rt.jar:1.8.0_31]

              at sun.nio.ch.SocketChannelImpl.read(Unknown Source) [rt.jar:1.8.0_31]

              at org.xnio.nio.NioSocketConduit.read(NioSocketConduit.java:282)

              at org.xnio.conduits.AbstractStreamSourceConduit.read(AbstractStreamSourceConduit.java:51)

              at io.undertow.conduits.ReadDataStreamSourceConduit.read(ReadDataStreamSourceConduit.java:67) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.conduits.FixedLengthStreamSourceConduit.read(FixedLengthStreamSourceConduit.java:239) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)

              at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:207) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2083) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.doParse(FormEncodedDataDefinition.java:133) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.parseBlocking(FormEncodedDataDefinition.java:251) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

              at io.undertow.servlet.spec.HttpServletRequestImpl.parseFormData(HttpServletRequestImpl.java:752) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

              ... 37 more

            • 3. Re: Wildfly 8.2: An existing connection was forcibly closed by the remote host
              jaikiran

              That looks odd. A few related questions. Are you sure it's a request parameter (i.e. part of the form fields or a query string) and not request attribute? Can you post the relevant form/HTML where this attribute is set. Anyway, the way it's failing is odd. Someone from the Undertow dev team might know more.

              • 4. Re: Wildfly 8.2: An existing connection was forcibly closed by the remote host
                ptuomaal

                We dropped the part of our application that was causing this in logs (we were already renewing that part). However the exception still comes time to time from other places from our app. This time the exception line has some sense because it comes from FilterChains doFilter method instead of our application code. So maybe this is exactly what the exception is trying to tell => remote connection was closed

                 

                -Petri

                 

                2015-04-24 09:38:28,396 ERROR [fi.elbit.skills.exception.ExceptionHandler] (default task-45) NTAjMjQwNDE1MDkzODI4LjM5Ng==: java.lang.RuntimeException: java.io.IOException: An existing connection was forcibly closed by the remote host

                  at io.undertow.servlet.spec.HttpServletRequestImpl.parseFormData(HttpServletRequestImpl.java:754) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.spec.HttpServletRequestImpl.getParameter(HttpServletRequestImpl.java:626) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at org.apache.myfaces.context.servlet.RequestParameterMap.getAttribute(RequestParameterMap.java:45) [myfaces-impl-2.1.15.jar:2.1.15]

                  at org.apache.myfaces.context.servlet.RequestParameterMap.getAttribute(RequestParameterMap.java:33) [myfaces-impl-2.1.15.jar:2.1.15]

                  at org.apache.myfaces.util.AbstractAttributeMap.containsKey(AbstractAttributeMap.java:62) [myfaces-impl-2.1.15.jar:2.1.15]

                  at org.apache.myfaces.renderkit.html.HtmlResponseStateManager.isPostback(HtmlResponseStateManager.java:303) [myfaces-impl-2.1.15.jar:2.1.15]

                  at org.apache.myfaces.lifecycle.DefaultRestoreViewSupport.isPostback(DefaultRestoreViewSupport.java:242) [myfaces-impl-2.1.15.jar:2.1.15]

                  at org.apache.myfaces.lifecycle.RestoreViewExecutor.execute(RestoreViewExecutor.java:120) [myfaces-impl-2.1.15.jar:2.1.15]

                  at org.apache.myfaces.lifecycle.LifecycleImpl.executePhase(LifecycleImpl.java:170) [myfaces-impl-2.1.15.jar:2.1.15]

                  at org.apache.myfaces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:117) [myfaces-impl-2.1.15.jar:2.1.15]

                  at javax.faces.webapp.FacesServlet.service(FacesServlet.java:197) [myfaces-api-2.1.15.jar:2.1.15]

                  at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at fi.elbit.skills.loginview.DefaultsFilter.doFilter(DefaultsFilter.java:117) [classes:]  ==> This line just calls chain.doFilter

                  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at org.apache.myfaces.webapp.filter.ExtensionsFilter.doFilter(ExtensionsFilter.java:357) [tomahawk20-1.1.14.jar:1.1.14]

                  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:61) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

                  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:56) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:45) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:63) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

                  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:261) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:247) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:76) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:166) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:759) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [rt.jar:1.8.0_31]

                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.8.0_31]

                  at java.lang.Thread.run(Unknown Source) [rt.jar:1.8.0_31]

                Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host

                  at sun.nio.ch.SocketDispatcher.read0(Native Method) [rt.jar:1.8.0_31]

                  at sun.nio.ch.SocketDispatcher.read(Unknown Source) [rt.jar:1.8.0_31]

                  at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) [rt.jar:1.8.0_31]

                  at sun.nio.ch.IOUtil.read(Unknown Source) [rt.jar:1.8.0_31]

                  at sun.nio.ch.SocketChannelImpl.read(Unknown Source) [rt.jar:1.8.0_31]

                  at org.xnio.nio.NioSocketConduit.read(NioSocketConduit.java:282)

                  at org.xnio.conduits.AbstractStreamSourceConduit.read(AbstractStreamSourceConduit.java:51)

                  at io.undertow.conduits.ReadDataStreamSourceConduit.read(ReadDataStreamSourceConduit.java:67) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.conduits.FixedLengthStreamSourceConduit.read(FixedLengthStreamSourceConduit.java:239) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)

                  at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:207) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2083) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.doParse(FormEncodedDataDefinition.java:133) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.parseBlocking(FormEncodedDataDefinition.java:251) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

                  at io.undertow.servlet.spec.HttpServletRequestImpl.parseFormData(HttpServletRequestImpl.java:752) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

                  ... 47 more