0 Replies Latest reply on Oct 21, 2014 10:00 AM by renan.polo.montebelo

Wildfly 8.1.0 Final using 100% CPU after SSL installation

renan.polo.montebelo Oct 21, 2014 10:00 AM

Hello,

I've seen some posts in the forum about people suffering from 100% CPU usage on Wildfly with SSL / TLS, but in all cases it's been resolved in earlier versions. Now it happens the same with me, but I'm already using 8.1.0 Final "Kenny".

The server starts OK, runs for hours OK, then some thread grabs 100% CPU of one core. Sometimes another thread does the same with another core and then I have near 100% CPU of my machine compromised. It's rare, but I've seen at least one case where the thread "recovered" and CPU utilization dropped to normal. BTW my normal CPU usage is around 10~20%. In my application an Android app communicates with Wildfly backend through JAX-RS. Some users are on 3G / Edge with high mobility through the city, and hence quite often I have some 'broken pipes' in my log, which I think is OK and never happens when on LAN. I think that might be the case where a bad connection made the SSL negotiation crazy and in a forever loop (just a guess).

If I undeploy my app, the thread still stucks at 100%. If I do a server reload (":reload"), things get back to normal.

With JConsole I was able to get the following trace of the hungry thread:

Name: default task-38

State: RUNNABLE

Total blocked: 6 Total waited: 208

Stack trace:

org.xnio.ssl.JsseSslConduitEngine.handleUnwrapResult(JsseSslConduitEngine.java:687)

- locked org.xnio.ByteBufferSlicePool$PooledByteBuffer@6fca55b3

org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:588)

- locked org.xnio.ByteBufferSlicePool$PooledByteBuffer@6fca55b3

org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:543)

org.xnio.ssl.JsseSslStreamSourceConduit.read(JsseSslStreamSourceConduit.java:89)

org.xnio.conduits.AbstractStreamSourceConduit.read(AbstractStreamSourceConduit.java:51)

io.undertow.conduits.ReadDataStreamSourceConduit.read(ReadDataStreamSourceConduit.java:49)

io.undertow.conduits.FixedLengthStreamSourceConduit.read(FixedLengthStreamSourceConduit.java:239)

org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)

io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:181)

io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:1952)

org.xnio.channels.Channels.readBlocking(Channels.java:294)

io.undertow.servlet.spec.ServletInputStreamImpl.readIntoBuffer(ServletInputStreamImpl.java:146)

io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:122)

com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:503)

com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding(ByteSourceJsonBootstrapper.java:129)

com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.constructParser(ByteSourceJsonBootstrapper.java:224)

com.fasterxml.jackson.core.JsonFactory._createParser(JsonFactory.java:1242)

com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:753)

com.fasterxml.jackson.jaxrs.base.ProviderBase._createParser(ProviderBase.java:791)

org.jboss.resteasy.plugins.providers.jackson.ResteasyJackson2Provider.readFrom(ResteasyJackson2Provider.java:111)

org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.readFrom(AbstractReaderInterceptorContext.java:59)

org.jboss.resteasy.core.interception.ServerReaderInterceptorContext.readFrom(ServerReaderInterceptorContext.java:62)

org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:51)

org.jboss.resteasy.security.doseta.DigitalVerificationInterceptor.aroundReadFrom(DigitalVerificationInterceptor.java:32)

org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:53)

org.jboss.resteasy.plugins.interceptors.encoding.GZIPDecodingInterceptor.aroundReadFrom(GZIPDecodingInterceptor.java:59)

org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:53)

org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:150)

org.jboss.resteasy.core.MethodInjectorImpl.injectArguments(MethodInjectorImpl.java:89)

org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:112)

org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)

org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)

org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237)

org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)

org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)

org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)

org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)

org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)

javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:61)

io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25)

io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:113)

io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:56)

io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25)

io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:45)

io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:61)

io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)

io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70)

io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)

io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25)

org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25)

io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:240)

io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)

io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73)

io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146)

io.undertow.server.Connectors.executeRootHandler(Connectors.java:177)

io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

java.lang.Thread.run(Thread.java:745)

Thanks for any help and let me know if I must gather further information.

Best Regards,

Renan

EDIT:

Just noticed when reloaded the server last time the following info in the log file, all related to the problematic thread. Couldn't understand what the information about the ClassLoader means. Obs: I'm not using JSF.

2014-10-21 13:51:26,743 WARN [org.jboss.resteasy.core.ExceptionHandler] (default task-38) Unknown exception while executing POST /Tracking/GeoLocation/add: java.lang.IllegalStateException: JBAS016071: Singleton not set for ModuleClassLoader for Module "deployment.MottomanBackend.war:main" from Service Module Loader. This means that you are trying to access a weld deployment with a Thread Context ClassLoader that is not associated with the deployment.

at org.jboss.as.weld.services.ModuleGroupSingletonProvider$TCCLSingleton.get(ModuleGroupSingletonProvider.java:77) [wildfly-weld-8.1.0.Final.jar:8.1.0.Final]

at org.jboss.as.weld.services.ModuleGroupSingletonProvider$TCCLSingleton.get(ModuleGroupSingletonProvider.java:134) [wildfly-weld-8.1.0.Final.jar:8.1.0.Final]

at org.jboss.weld.Container.instance(Container.java:65) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]

at org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:75) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]

at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:99) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]

at org.jboss.weld.proxies.ValidatorFactory$627848863$Proxy$_$$_WeldClientProxy.getValidator(Unknown Source) [weld-core-impl-2.1.2.Final.jar:]

at org.jboss.as.ee.beanvalidation.LazyValidatorFactory.getValidator(LazyValidatorFactory.java:73)

at org.jboss.resteasy.plugins.validation.GeneralValidatorImpl.getValidator(GeneralValidatorImpl.java:556) [resteasy-validator-provider-11-3.0.8.Final.jar:]

at org.jboss.resteasy.plugins.validation.GeneralValidatorImpl.validateAllParameters(GeneralValidatorImpl.java:143) [resteasy-validator-provider-11-3.0.8.Final.jar:]

at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:116) [resteasy-jaxrs-3.0.8.Final.jar:]

at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296) [resteasy-jaxrs-3.0.8.Final.jar:]

at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250) [resteasy-jaxrs-3.0.8.Final.jar:]

at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237) [resteasy-jaxrs-3.0.8.Final.jar:]

at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356) [resteasy-jaxrs-3.0.8.Final.jar:]

at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179) [resteasy-jaxrs-3.0.8.Final.jar:]

at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) [resteasy-jaxrs-3.0.8.Final.jar:]

at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.0.8.Final.jar:]

at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.0.8.Final.jar:]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]

at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:61) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:113) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:56) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:45) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:61) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:240) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65]

at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65]

2014-10-21 13:51:26,806 ERROR [io.undertow.request] (default task-38) UT005023: Exception handling request to /mottoman/app/Tracking/GeoLocation/add: org.jboss.resteasy.spi.UnhandledException: java.lang.IllegalStateException: JBAS016071: Singleton not set for ModuleClassLoader for Module "deployment.MottomanBackend.war:main" from Service Module Loader. This means that you are trying to access a weld deployment with a Thread Context ClassLoader that is not associated with the deployment.

at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:247) [resteasy-jaxrs-3.0.8.Final.jar:]

at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:149) [resteasy-jaxrs-3.0.8.Final.jar:]

at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:372) [resteasy-jaxrs-3.0.8.Final.jar:]