1 Reply Latest reply on Aug 4, 2019 1:37 AM by mmuthu

    Wildfly 10.1 Hanging issue

    mmuthu

      We have an application running with Wildfly 10.1 and Java 8 Update 152. The application takes standard SAML response to implement login. We staged the application and tested before we rollout to production. Once we rolled out to production, we do see that client who tested in stage is unable to make login call. All the call from client to do login goes into hung state where as if we call same login API from our own application it goes through with out issue. Occasionally we do see login success from client application as well but that is very rare. Following is the trace when it hung.

       

      "default task-55" #282 prio=5 os_prio=0 tid=0x00007faed0007000 nid=0x377 runnable [0x00007fad8efee000]

         java.lang.Thread.State: RUNNABLE

                      at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

                      at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

                      at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

                      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

                      - locked <0x00000006643b4690> (a sun.nio.ch.Util$3)

                      - locked <0x00000006643b4680> (a java.util.Collections$UnmodifiableSet)

                      - locked <0x00000006643b4568> (a sun.nio.ch.EPollSelectorImpl)

                      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

                      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

                      at org.xnio.nio.SelectorUtils.await(SelectorUtils.java:51)

                      at org.xnio.nio.NioSocketConduit.awaitReadable(NioSocketConduit.java:358)

                      at io.undertow.protocols.ssl.SslConduit.awaitReadable(SslConduit.java:312)

                      at org.xnio.conduits.AbstractSourceConduit.awaitReadable(AbstractSourceConduit.java:66)

                      at io.undertow.conduits.ReadDataStreamSourceConduit.awaitReadable(ReadDataStreamSourceConduit.java:101)

                      at io.undertow.conduits.FixedLengthStreamSourceConduit.awaitReadable(FixedLengthStreamSourceConduit.java:285)

                      at org.xnio.conduits.ConduitStreamSourceChannel.awaitReadable(ConduitStreamSourceChannel.java:151)

                      at io.undertow.channels.DetachableStreamSourceChannel.awaitReadable(DetachableStreamSourceChannel.java:77)

                      at io.undertow.server.HttpServerExchange$ReadDispatchChannel.awaitReadable(HttpServerExchange.java:2126)

                      at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.parseBlocking(FormEncodedDataDefinition.java:254)

                      at io.undertow.servlet.spec.HttpServletRequestImpl.parseFormData(HttpServletRequestImpl.java:761)

                      at io.undertow.servlet.spec.HttpServletRequestImpl.getParameterValues(HttpServletRequestImpl.java:688)

                      at org.springframework.web.context.request.ServletWebRequest.getParameterValues(ServletWebRequest.java:159)

                      at org.springframework.web.method.annotation.RequestParamMethodArgumentResolver.resolveName(RequestParamMethodArgumentResolver.java:173)

                      at org.springframework.web.method.annotation.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:103)

                      at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:121)

                      at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:158)

                      at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:128)

                      at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)

                      at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849)

                      at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760)

                      at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)

                      at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)

                      at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)

                      at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)

                      at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)

                      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

                      at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)

                      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

                      at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

                      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)

                      at dapi.filters.CORSFilter.doFilter(CORSFilter.java:57)

                      at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

                      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

                      at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)

                      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

                      at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

                      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

                      at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

                      at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

                      at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

                      at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

                      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                      at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

                      at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

                      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                      at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)

                      at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

                      at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

                      at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)

                      at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

                      at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

                      at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

                      at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

                      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                      at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

                      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                      at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)

                      at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)

                      at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)

                      at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)

                      at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)

                      at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)

                      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

                      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

                      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

                      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

                      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

                      at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)

                      at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

                      at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)

                      at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

                      at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

                      at java.lang.Thread.run(Thread.java:748)

       

      I have been running with this issue for few days with out success. WIldfly 10.1 is not something new to us, we have been using in our production environment for last two years. When we upgraded to 10.1, we did upgraded NIO libraries to following version and our undertow version is 1.4.0.Final.

       

      ./system/layers/base/org/jboss/xnio/main/xnio-api-3.4.6.Final.jar

      ./system/layers/base/org/jboss/xnio/nio/main/xnio-nio-3.4.3.Final.jar

       

      Any clue?

       

       

        • 1. Re: Wildfly 10.1 Hanging issue
          mmuthu

          I enabled TRACE level logs in standalone.xml and see following in the log when client makes a call. Does it yield any clue for any one. It suggest me that there is some thing wrong in SSL certification layer but still looking out for the detailed RCA for the problem.

           

          TRACE [org.xnio.nio.selector] (default I/O-23) Beginning select on sun.nio.ch.EPollSelectorImpl@327d828a (with timeout)

          TRACE [org.xnio.nio.selector] (default I/O-23) Selected on sun.nio.ch.EPollSelectorImpl@327d828a

          TRACE [org.xnio.nio] (default I/O-23) Running task io.undertow.server.protocol.ParseTimeoutUpdater@77bbf0c

          TRACE [org.xnio.nio.selector] (default I/O-23) Beginning select on sun.nio.ch.EPollSelectorImpl@327d828a

          INFO  [stdout] (default task-18) default task-18, called closeInbound()

          INFO  [stdout] (default task-18) default task-18, called closeInbound()

          INFO  [stdout] (default task-18) default task-18, fatal error: 80: Inbound closed before receiving peer's close_notify: possible truncation attack?

          INFO  [stdout] (default task-18) default task-18, fatal error: 80: Inbound closed before receiving peer's close_notify: possible truncation attack?

          INFO  [stdout] (default task-18) javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?

          INFO  [stdout] (default task-18) javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?

          INFO  [stdout] (default task-18) %% Invalidated:  [Session-8, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]

          INFO  [stdout] (default task-18) %% Invalidated:  [Session-8, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]

          INFO  [stdout] (default task-18) default task-18, SEND TLSv1.2 ALERT:  fatal, description = internal_error

          INFO  [stdout] (default task-18) default task-18, SEND TLSv1.2 ALERT:  fatal, description = internal_error

          INFO  [stdout] (default task-18) Padded plaintext before ENCRYPTION:  len = 80

          NFO  [stdout] (default task-18) Padded plaintext before ENCRYPTION:  len = 80