1 Reply Latest reply on Jul 3, 2017 2:59 AM by petermkruse

    WildFly 10.1: UT000128: Remote peer closed connection before all data could be read

    petermkruse

      We’re experiencing a very weird situation where WildFly (or perhaps more precisely, undertow) crashes when receiving a standard xml request via a web-service, from a specific mainframe. As it apparently misinterprets the client sending a TCP package stating it has completed sending the xml message as a signal the session must terminate immediately. Looks like some very specific error in undertow that only occurs under very special circumstances. We cannot find reports of an identical issue and are hoping someone has a good idea to fix the issue. The old server is running on jBoss 5.1.0.GA and the new server is running on WildFly 10.0.0 (patched with 10.1.0.Final)

       

      The scenario is this: Our client has an old web-server and a newer one built to replace the old one, with newer OS and slightly updated application. The old server is running production while we cannot upgrade to the new server due to the crash issue. Please note that it is only the request from a specific mainframe that makes the session crash. Another mainframe and PC clients with a dedicated client works just fine towards both the old and the new server.

       

      Specifically, what happens is this: A mainframe running VM OS sends a xml request to the web-service of a webserver running WildFly/undertow/java/IIS/Windows Server. The communication protocol is standard http on port 8080, on the internal network. The mainframe seems to be running in asynchronous mode so when it has transferred the xml it sends a [FIN,ACK] TCP package to inform the server it has completed the transfer, and waits for a response.

      When the old webserver receives the xml request it generates a correct response and returns this, and then asks to close the session. All is fine. Unclear whether it works in asynchronous or synchronous mode but it responds correctly.

      With the new server the traffic is identical and the [FIN,ACK] TCP package is again sent immediately after the xml message has been sent. Apparently the [FIN,ACK] TCP package is received on the server before interpretation of the final line in the xml document has completed, and instead of reading the final line - as it should - the session decides to do a hara-kiri and crashes with this error message:

      1. java.io.IOException: UT000128: Remote peer closed connection before all data could be read

       

      From the network and server logs it seems to be the [FIN,ACK] package that crashes the session on the new server. The response is consistent over a number of tests.

      Please see error logs below.

       

      In the log the below can be seen on the old server:

      Enter: DeserializationContext::endPrefixMapping(xsi)

      Exit: DeserializationContext::endPrefixMapping()

      Enter: DeserializationContext::endDocument()

      Exit: DeserializationContext::endDocument()

       

      - the last two lines shows it reading the last line of the xml. These two lines are missing in the logs from the new server.

       

       

      While on the new server this happens:

      Enter: DeserializationContext::endPrefixMapping(xsi)

      Exit: DeserializationContext::endPrefixMapping()

      UT005007: Request was not fully consumed

      UT005007: Request was not fully consumed

      • org.apache.axis.i18n.resource::handleGetObject(axisFault00)

      AxisFault:: java.io.IOException: UT000128: Remote peer closed connection before all data could be read

      at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)

      at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:701)

      at org.apache.axis.Message.getSOAPEnvelope(Message.java:435)

      at org.apache.axis.server.AxisServer.initSOAPConstants(AxisServer.java:345)

      at org.apache.axis.server.AxisServer.invoke(AxisServer.java:279)

      at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

      at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

      at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

      at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

      at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

      at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

      at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

      at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)

      at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

      at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

      at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)

      at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

      at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

      at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

      at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)

      at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)

      at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)

      at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)

      at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)

      at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)

      at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

      at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)

      at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

      at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

      at java.lang.Thread.run(Thread.java:745)

      Caused by: java.io.IOException: UT000128: Remote peer closed connection before all data could be read

      at io.undertow.conduits.FixedLengthStreamSourceConduit.exitRead(FixedLengthStreamSourceConduit.java:338)

      at io.undertow.conduits.FixedLengthStreamSourceConduit.read(FixedLengthStreamSourceConduit.java:255)

      at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)

      at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:209)

      at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2265)

      at org.xnio.channels.Channels.readBlocking(Channels.java:294)

      at io.undertow.servlet.spec.ServletInputStreamImpl.readIntoBuffer(ServletInputStreamImpl.java:168)

      at io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:144)

      at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(XMLEntityManager.java:3147)

      at org.apache.xerces.impl.io.UTF8Reader.read(UTF8Reader.java:306)

      at org.apache.xerces.impl.XMLEntityScanner.load(XMLEntityScanner.java:1753)

      at org.apache.xerces.impl.XMLEntityScanner.skipSpaces(XMLEntityScanner.java:1304)

      at org.apache.xerces.impl.XMLDocumentScannerImpl$TrailingMiscDispatcher.dispatch(XMLDocumentScannerImpl.java:1239)

      at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:324)

      at org.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:875)

      at org.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:798)

      at org.apache.xerces.parsers.XMLParser.parse(XMLParser.java:108)

      at org.apache.xerces.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1198)

      at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:564)

      at org.apache.xerces.jaxp.SAXParserImpl.parse(SAXParserImpl.java:298)

      at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)

      at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)

      ... 47 more

       

      After which it starts creating a nicely formatted xml error report and returns this.

       

       

      Network dump – From the old server where it works: attached image network-trace-old.png

         

       

      Network dump – From the new server where it doesn’t work: attached image network-trace-new.png

         

       

      PS: The server on which it works reports it is using http/1.1 whereas the one on which is doesn’t work reports it is using http/1.0. This is a leftover from our own testing where we tried to switch between the two types, but it made no difference.

       

       

      Question:

      Is there any way to make undertow continue and avoid the exceptions UT005007 / UT000128 ?