1 Reply Latest reply on Feb 13, 2017 8:35 AM by Alexander Wagner

    Wildfly hanging threads on ConduitStreamSinkChannel.awaitWritable

    Alexander Wagner Newbie

      Hello forum users,

       

      we have currently a big issue and we don't know how to solve it. It occurs only in production environment so far. After a while we have high CPU load which comes from a view hanging threads. Until we have to restart the Wildfly because no requests were answered anymore we only see one to four threads hanging at the same  point:

       

      org.xnio.conduits.ConduitStreamSinkChannel.awaitWritable(ConduitStreamSinkChannel.java:134)
      

       

      In the end we do not need to restart the Wildfly because of the CPU load, but after a while of the hanging thread(s) we can see a lot, up to 1000, of open TCP sockets in the state CLOSE_WAIT which targeting the wildfly port 80 and 443. We only guess that this reason why no more request are answered. Maybe there is no relation between the hanging threads and this open sockets but it happens always in this order.

       

      We would be very thankful for every advice!

       

      We detected the CPU usage of the threads via the LWP id

       

      Environment

      OS

      Linux lvpsXXXX.dedicated.hosteurope.de 2.6.32-028stab101.1 #1 SMP Sun Jun 24 19:50:48 MSD 2012 x86_64 GNU/Linux

      Java

      java version "1.8.0_112"

      Java(TM) SE Runtime Environment (build 1.8.0_112-b15)

      Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

      WildFly

      wildfly-10.1.0.Final

       

      Thread Dumps

      "default task-10" #129 prio=5 os_prio=0 tid=0x00002ae8e86a6800 nid=0x6fe runnable [0x00002ae8ef9f9000]
         java.lang.Thread.State: RUNNABLE
          at org.xnio.conduits.ConduitStreamSinkChannel.awaitWritable(ConduitStreamSinkChannel.java:134)
          at io.undertow.channels.DetachableStreamSinkChannel.awaitWritable(DetachableStreamSinkChannel.java:87)
          at io.undertow.server.HttpServerExchange$WriteDispatchChannel.awaitWritable(HttpServerExchange.java:1962)
          at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:566)
          at io.undertow.servlet.spec.ServletOutputStreamImpl.flushInternal(ServletOutputStreamImpl.java:478)
          at io.undertow.servlet.spec.ServletOutputStreamImpl.flush(ServletOutputStreamImpl.java:465)
          at io.undertow.servlet.spec.ServletPrintWriter.flush(ServletPrintWriter.java:72)
          at io.undertow.servlet.spec.ServletPrintWriterDelegate.flush(ServletPrintWriterDelegate.java:94)
          at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:231)
          at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147)
          at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111)
          at org.ocpsoft.rewrite.servlet.impl.HttpRewriteResultHandler.handleResult(HttpRewriteResultHandler.java:42)
          at org.ocpsoft.rewrite.servlet.RewriteFilter.rewrite(RewriteFilter.java:297)
          at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:198)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
          at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
          at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
          at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
          at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
          at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
          at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
          at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:104)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
          at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
          at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
          at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
          at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
          at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
          at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
          at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
          at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
          at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
          at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
          at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
          at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
          at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
          at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
          at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
          at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
          at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      
      "default task-4" #108 prio=5 os_prio=0 tid=0x00002ae8e0be6000 nid=0x6e3 runnable [0x00002ae8edead000]
         java.lang.Thread.State: RUNNABLE
          at org.xnio.conduits.ConduitStreamSinkChannel.awaitWritable(ConduitStreamSinkChannel.java:134)
          at io.undertow.channels.DetachableStreamSinkChannel.awaitWritable(DetachableStreamSinkChannel.java:87)
          at io.undertow.server.HttpServerExchange$WriteDispatchChannel.awaitWritable(HttpServerExchange.java:1962)
          at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:566)
          at io.undertow.servlet.spec.ServletOutputStreamImpl.flushInternal(ServletOutputStreamImpl.java:478)
          at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:162)
          at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:245)
          at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:249)
          at io.undertow.servlet.spec.ServletPrintWriterDelegate.write(ServletPrintWriterDelegate.java:119)
          at com.sun.faces.application.view.WriteBehindStateWriter.flushToWriter(WriteBehindStateWriter.java:287)
          at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:475)
          at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:134)
          at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337)
          at org.ocpsoft.rewrite.faces.RewriteViewHandler.renderView(RewriteViewHandler.java:196)
          at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337)
          at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337)
          at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337)
          at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:120)
          at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
          at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:219)
          at org.apache.deltaspike.jsf.impl.listener.request.DeltaSpikeLifecycleWrapper.render(DeltaSpikeLifecycleWrapper.java:111)
          at javax.faces.lifecycle.LifecycleWrapper.render(LifecycleWrapper.java:92)
          at org.apache.deltaspike.jsf.impl.listener.request.JsfClientWindowAwareLifecycleWrapper.render(JsfClientWindowAwareLifecycleWrapper.java:160)
          at javax.faces.webapp.FacesServlet.service(FacesServlet.java:659)
          at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
          at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
          at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274)
          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209)
          at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221)
          at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147)
          at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111)
          at org.ocpsoft.rewrite.servlet.impl.HttpRewriteResultHandler.handleResult(HttpRewriteResultHandler.java:42)
          at org.ocpsoft.rewrite.servlet.RewriteFilter.rewrite(RewriteFilter.java:297)
          at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:198)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
          at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
          at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
          at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
          at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
          at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
          at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
          at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:104)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
          at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
          at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
          at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
          at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
          at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
          at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
          at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
          at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
          at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
          at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
          at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
          at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
          at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
          at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
          at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
          at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
          at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      
      "default task-62" #250 prio=5 os_prio=0 tid=0x00002ae8e9157000 nid=0xc28 runnable [0x00002ae8fe0c9000]
         java.lang.Thread.State: RUNNABLE
          at org.xnio.conduits.ConduitStreamSinkChannel.awaitWritable(ConduitStreamSinkChannel.java:134)
          at io.undertow.channels.DetachableStreamSinkChannel.awaitWritable(DetachableStreamSinkChannel.java:87)
          at io.undertow.server.HttpServerExchange$WriteDispatchChannel.awaitWritable(HttpServerExchange.java:1962)
          at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:566)
          at io.undertow.servlet.spec.ServletOutputStreamImpl.flushInternal(ServletOutputStreamImpl.java:478)
          at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:162)
          at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:245)
          at io.undertow.servlet.spec.ServletPrintWriterDelegate.write(ServletPrintWriterDelegate.java:114)
          at com.sun.faces.application.view.WriteBehindStateWriter.write(WriteBehindStateWriter.java:147)
          at com.sun.faces.renderkit.html_basic.HtmlResponseWriter.flushAttributes(HtmlResponseWriter.java:1211)
          at com.sun.faces.renderkit.html_basic.HtmlResponseWriter.endElement(HtmlResponseWriter.java:588)
          at org.primefaces.component.selectbooleanbutton.SelectBooleanButtonRenderer.encodeMarkup(SelectBooleanButtonRenderer.java:113)
          at org.primefaces.component.selectbooleanbutton.SelectBooleanButtonRenderer.encodeEnd(SelectBooleanButtonRenderer.java:56)
          at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:920)
          at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1863)
          at org.primefaces.component.datatable.DataTableRenderer.encodeFilter(DataTableRenderer.java:731)
          at org.primefaces.component.datatable.DataTableRenderer.encodeColumnHeader(DataTableRenderer.java:535)
          at org.primefaces.component.datatable.DataTableRenderer.encodeThead(DataTableRenderer.java:818)
          at org.primefaces.component.datatable.DataTableRenderer.encodeThead(DataTableRenderer.java:788)
          at org.primefaces.component.datatable.DataTableRenderer.encodeScrollableTable(DataTableRenderer.java:386)
          at org.primefaces.component.datatable.DataTableRenderer.encodeMarkup(DataTableRenderer.java:283)
          at org.primefaces.component.datatable.DataTableRenderer.encodeEnd(DataTableRenderer.java:92)
          at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:920)
          at org.primefaces.renderkit.CoreRenderer.renderChild(CoreRenderer.java:89)
          at org.primefaces.renderkit.CoreRenderer.renderChildren(CoreRenderer.java:72)
          at org.primefaces.component.outputpanel.OutputPanelRenderer.encodeMarkup(OutputPanelRenderer.java:65)
          at org.primefaces.component.outputpanel.OutputPanelRenderer.encodeEnd(OutputPanelRenderer.java:41)
          at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:920)
          at org.primefaces.renderkit.CoreRenderer.renderChild(CoreRenderer.java:89)
          at org.primefaces.renderkit.CoreRenderer.renderChildren(CoreRenderer.java:72)
          at org.primefaces.component.outputpanel.OutputPanelRenderer.encodeMarkup(OutputPanelRenderer.java:65)
          at org.primefaces.component.outputpanel.OutputPanelRenderer.encodeEnd(OutputPanelRenderer.java:41)
          at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:920)
          at org.primefaces.renderkit.CoreRenderer.renderChild(CoreRenderer.java:89)
          at org.primefaces.renderkit.CoreRenderer.renderChildren(CoreRenderer.java:72)
          at org.primefaces.component.outputpanel.OutputPanelRenderer.encodeMarkup(OutputPanelRenderer.java:65)
          at org.primefaces.component.outputpanel.OutputPanelRenderer.encodeEnd(OutputPanelRenderer.java:41)
          at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:920)
          at org.primefaces.renderkit.CoreRenderer.renderChild(CoreRenderer.java:89)
          at org.primefaces.renderkit.CoreRenderer.renderChildren(CoreRenderer.java:72)
          at org.primefaces.component.outputpanel.OutputPanelRenderer.encodeMarkup(OutputPanelRenderer.java:65)
          at org.primefaces.component.outputpanel.OutputPanelRenderer.encodeEnd(OutputPanelRenderer.java:41)
          at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:920)
          at org.primefaces.renderkit.CoreRenderer.renderChild(CoreRenderer.java:89)
          at org.primefaces.renderkit.CoreRenderer.renderChildren(CoreRenderer.java:72)
          at org.primefaces.component.outputpanel.OutputPanelRenderer.encodeMarkup(OutputPanelRenderer.java:65)
          at org.primefaces.component.outputpanel.OutputPanelRenderer.encodeEnd(OutputPanelRenderer.java:41)
          at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:920)
          at org.primefaces.renderkit.CoreRenderer.renderChild(CoreRenderer.java:89)
          at org.primefaces.renderkit.CoreRenderer.renderChildren(CoreRenderer.java:72)
          at org.primefaces.component.outputpanel.OutputPanelRenderer.encodeMarkup(OutputPanelRenderer.java:65)
          at org.primefaces.component.outputpanel.OutputPanelRenderer.encodeEnd(OutputPanelRenderer.java:41)
          at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:920)
          at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1863)
          at javax.faces.render.Renderer.encodeChildren(Renderer.java:176)
          at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:890)
          at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1856)
          at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859)
          at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859)
          at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:458)
          at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:134)
          at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337)
          at org.ocpsoft.rewrite.faces.RewriteViewHandler.renderView(RewriteViewHandler.java:196)
          at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337)
          at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337)
          at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337)
          at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:120)
          at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
          at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:219)
          at org.apache.deltaspike.jsf.impl.listener.request.DeltaSpikeLifecycleWrapper.render(DeltaSpikeLifecycleWrapper.java:111)
          at javax.faces.lifecycle.LifecycleWrapper.render(LifecycleWrapper.java:92)
          at org.apache.deltaspike.jsf.impl.listener.request.JsfClientWindowAwareLifecycleWrapper.render(JsfClientWindowAwareLifecycleWrapper.java:160)
          at javax.faces.webapp.FacesServlet.service(FacesServlet.java:659)
          at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
          at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
          at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274)
          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209)
          at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221)
          at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147)
          at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111)
          at org.ocpsoft.rewrite.servlet.impl.HttpRewriteResultHandler.handleResult(HttpRewriteResultHandler.java:42)
          at org.ocpsoft.rewrite.servlet.RewriteFilter.rewrite(RewriteFilter.java:297)
          at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:198)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
          at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
          at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
          at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
          at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
          at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
          at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
          at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
          at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
          at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:104)
          at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
          at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
          at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
          at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
          at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
          at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
          at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
          at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
          at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
          at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
          at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
          at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
          at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
          at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
          at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
          at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
          at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
          at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
          at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
          at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)