StackOverflowError when JBoss Modules defines a class

[paul.robinson]

Hello,

I'm, intermittently, seeing the following stacktrace when invoking a Web Service deployed to AS8 master (started last week and is still happening with today's code). The problem seems to occur when JBoss Modules tries to define a class from within the "org.jboss.ws.common:main" module. The issue occurs in a test that has been consistently passing for over a year. The issue seems to coincide with updates to JBossWS, CXF and the inclusion of Undertow. As this issue is intermittent, it's hard to pinpoint the exact set of commits that seem to have affected this.

Any ideas?

Cheers,

Paul.

[33m20:31:45,823 WARN  [org.jboss.modules] (default task-1) Failed to define class org.jboss.ws.common.reflection.AbstractClassProcessor in Module "org.jboss.ws.common:main" from local module loader @bc713e (finder: local module finder @1aea1ed (roots: /home/hudson/workspace/narayana/jboss-as/build/target/jboss-as-8.0.0.Alpha1-SNAPSHOT/modules,/home/hudson/workspace/narayana/jboss-as/build/target/jboss-as-8.0.0.Alpha1-SNAPSHOT/modules/system/layers/base)): java.lang.StackOverflowError
 at java.lang.ClassLoader.defineClass1(Native Method) [rt.jar:1.7.0_09]
 at java.lang.ClassLoader.defineClass(ClassLoader.java:791) [rt.jar:1.7.0_09]
 at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:338) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:416) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:254) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:73) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.Module.loadModuleClass(Module.java:524) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:182) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:374) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119) [jboss-modules.jar:1.2.0.CR2]
 at java.lang.ClassLoader.defineClass1(Native Method) [rt.jar:1.7.0_09]
 at java.lang.ClassLoader.defineClass(ClassLoader.java:791) [rt.jar:1.7.0_09]
 at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:338) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:416) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:254) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:73) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.Module.loadModuleClass(Module.java:524) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:182) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:374) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119) [jboss-modules.jar:1.2.0.CR2]
 at java.lang.ClassLoader.defineClass1(Native Method) [rt.jar:1.7.0_09]
 at java.lang.ClassLoader.defineClass(ClassLoader.java:791) [rt.jar:1.7.0_09]
 at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:338) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:416) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:254) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:73) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.Module.loadModuleClass(Module.java:524) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:182) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:374) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119) [jboss-modules.jar:1.2.0.CR2]
 at java.lang.ClassLoader.defineClass1(Native Method) [rt.jar:1.7.0_09]
 at java.lang.ClassLoader.defineClass(ClassLoader.java:791) [rt.jar:1.7.0_09]
 at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:338) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:416) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:254) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:73) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.Module.loadModuleClass(Module.java:524) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:182) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:374) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119) [jboss-modules.jar:1.2.0.CR2]
 at java.lang.ClassLoader.defineClass1(Native Method) [rt.jar:1.7.0_09]
 at java.lang.ClassLoader.defineClass(ClassLoader.java:791) [rt.jar:1.7.0_09]
 at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:338) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:416) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:254) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:73) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.Module.loadModuleClass(Module.java:524) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:182) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:374) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119) [jboss-modules.jar:1.2.0.CR2]
 at java.lang.ClassLoader.defineClass1(Native Method) [rt.jar:1.7.0_09]
 at java.lang.ClassLoader.defineClass(ClassLoader.java:791) [rt.jar:1.7.0_09]
 at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:338) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:416) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:254) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:73) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.Module.loadModuleClass(Module.java:524) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:182) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:374) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119) [jboss-modules.jar:1.2.0.CR2]
 at org.jboss.ws.common.injection.InjectionHelper.<clinit>(InjectionHelper.java:43)
 at org.jboss.wsf.stack.cxf.transport.ServletHelper.injectServiceAndHandlerResources(ServletHelper.java:121)
 at org.jboss.wsf.stack.cxf.transport.ServletHelper.initServiceEndpoint(ServletHelper.java:101)
 at org.jboss.wsf.stack.cxf.transport.ServletHelper.initEndpoint(ServletHelper.java:71)
 at org.jboss.wsf.stack.cxf.CXFServletExt.loadBus(CXFServletExt.java:67)
 at org.jboss.wsf.stack.cxf.CXFServletExt.init(CXFServletExt.java:61)
 at org.jboss.wsf.spi.deployment.WSFServlet.init(WSFServlet.java:57) [jbossws-spi-2.2.0.Alpha1.jar:2.2.0.Alpha1]
 at io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:145)
 at io.undertow.servlet.core.ManagedServlet.getServlet(ManagedServlet.java:97)
 at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:87)
 at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:39)
 at org.jboss.as.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:76)
 at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
 at io.undertow.server.HttpHandlers.executeHandler(HttpHandlers.java:46)
 at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:52)
 at io.undertow.server.HttpHandlers.executeHandler(HttpHandlers.java:46)
 at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
 at io.undertow.server.HttpHandlers.executeHandler(HttpHandlers.java:46)
 at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:66)
 at io.undertow.server.HttpHandlers.executeHandler(HttpHandlers.java:46)
 at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:51)
 at io.undertow.server.HttpHandlers.executeHandler(HttpHandlers.java:46)
 at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:71)
 at io.undertow.server.HttpHandlers.executeHandler(HttpHandlers.java:46)
 at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:26)
 at org.jboss.as.undertow.security.SecurityContextCreationHandler.handleRequest(SecurityContextCreationHandler.java:32)
 at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:119)
 at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:107)
 at io.undertow.servlet.handlers.ServletInitialHandler.handleRequest(ServletInitialHandler.java:82)
 at io.undertow.server.HttpHandlers.executeRootHandler(HttpHandlers.java:53)
 at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:501)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09]
 at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09]

[paul.robinson]

Also, we are seeing a similar issue on one of the AS8 JAX-RS smoke tests:

testsuite/integration/smoke/target/surefire-reports/org.jboss.as.test.smoke.jaxrs.JaxrsTestCase

See bellow:

[0m [33m00:37:24,181 WARN  [org.jboss.modules] (default task-1) Failed to define class org.codehaus.jackson.map.JsonSerializableWithType in Module "org.codehaus.jackson.jackson-mapper-asl:main" from local module loader @b7e998 (finder: local module finder @4904c8 (roots: /home/hudson/workspace/narayana/jboss-as/testsuite/integration/smoke/../../../build/target/jboss-as-8.0.0.Alpha1-SNAPSHOT/modules,/home/hudson/workspace/narayana/jboss-as/testsuite/integration/smoke/../../../build/target/jboss-as-8.0.0.Alpha1-SNAPSHOT/modules/system/layers/base,/home/hudson/workspace/narayana/jboss-as/testsuite/integration/smoke/target/modules)): java.lang.StackOverflowError
          at java.util.regex.Pattern$5.isSatisfiedBy(Pattern.java:5151) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$5.isSatisfiedBy(Pattern.java:5151) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$Branch.match(Pattern.java:4500) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$Branch.match(Pattern.java:4500) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$Branch.match(Pattern.java:4500) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$Curly.match0(Pattern.java:4177) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$Curly.match(Pattern.java:4132) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$Branch.match(Pattern.java:4502) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$Branch.match(Pattern.java:4500) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3715) [rt.jar:1.7.0_09]
          at java.util.regex.Pattern$Start.match(Pattern.java:3408) [rt.jar:1.7.0_09]
          at java.util.regex.Matcher.search(Matcher.java:1199) [rt.jar:1.7.0_09]
          at java.util.regex.Matcher.find(Matcher.java:618) [rt.jar:1.7.0_09]
          at java.util.Formatter.parse(Formatter.java:2517) [rt.jar:1.7.0_09]
          at java.util.Formatter.format(Formatter.java:2469) [rt.jar:1.7.0_09]
          at java.util.Formatter.format(Formatter.java:2423) [rt.jar:1.7.0_09]
          at java.lang.String.format(String.java:2797) [rt.jar:1.7.0_09]
          at org.jboss.modules.log.JDKModuleLogger.classDefineFailed(JDKModuleLogger.java:194) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:426) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:254) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:73) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.Module.loadModuleClass(Module.java:524) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:182) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:374) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119) [jboss-modules.jar:1.2.0.CR2]
          at java.lang.ClassLoader.defineClass1(Native Method) [rt.jar:1.7.0_09]
          at java.lang.ClassLoader.defineClass(ClassLoader.java:791) [rt.jar:1.7.0_09]
          at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:338) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:416) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:254) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:73) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.Module.loadModuleClass(Module.java:524) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:182) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:374) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119) [jboss-modules.jar:1.2.0.CR2]
          at java.lang.ClassLoader.defineClass1(Native Method) [rt.jar:1.7.0_09]
          at java.lang.ClassLoader.defineClass(ClassLoader.java:791) [rt.jar:1.7.0_09]
          at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:338) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:416) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:254) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:73) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.Module.loadModuleClass(Module.java:524) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:182) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:374) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119) [jboss-modules.jar:1.2.0.CR2]
          at java.lang.ClassLoader.defineClass1(Native Method) [rt.jar:1.7.0_09]
          at java.lang.ClassLoader.defineClass(ClassLoader.java:791) [rt.jar:1.7.0_09]
          at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:338) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:416) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:254) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:73) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.Module.loadModuleClass(Module.java:524) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:182) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:374) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119) [jboss-modules.jar:1.2.0.CR2]
          at java.lang.ClassLoader.defineClass1(Native Method) [rt.jar:1.7.0_09]
          at java.lang.ClassLoader.defineClass(ClassLoader.java:791) [rt.jar:1.7.0_09]
          at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:338) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:416) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:254) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:73) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.Module.loadModuleClass(Module.java:524) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:182) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:374) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119) [jboss-modules.jar:1.2.0.CR2]
          at org.jboss.resteasy.plugins.providers.jackson.JacksonJsonpInterceptor.<clinit>(JacksonJsonpInterceptor.java:74) [resteasy-jackson-provider-3.0-beta-3.jar:]
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_09]
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_09]
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_09]
          at java.lang.reflect.Constructor.newInstance(Constructor.java:525) [rt.jar:1.7.0_09]
          at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:132) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.spi.ResteasyProviderFactory.injectedInstance(ResteasyProviderFactory.java:1832) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$AbstractInterceptorFactory.createInterceptor(JaxrsInterceptorRegistry.java:156) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$OnDemandInterceptorFactory.initialize(JaxrsInterceptorRegistry.java:174) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$OnDemandInterceptorFactory.checkInitialize(JaxrsInterceptorRegistry.java:189) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$OnDemandInterceptorFactory.getInterceptor(JaxrsInterceptorRegistry.java:200) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$AbstractInterceptorFactory.postMatch(JaxrsInterceptorRegistry.java:137) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry.postMatch(JaxrsInterceptorRegistry.java:377) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.core.ResourceMethod.<init>(ResourceMethod.java:142) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.core.ResourceMethodRegistry.processMethod(ResourceMethodRegistry.java:280) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.core.ResourceMethodRegistry.addResourceFactory(ResourceMethodRegistry.java:129) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.core.ResourceMethodRegistry.addResourceFactory(ResourceMethodRegistry.java:111) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.core.ResourceMethodRegistry.addResourceFactory(ResourceMethodRegistry.java:88) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.core.ResourceMethodRegistry.addPerRequestResource(ResourceMethodRegistry.java:77) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.spi.ResteasyDeployment.registration(ResteasyDeployment.java:370) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:226) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:67) [resteasy-jaxrs-3.0-beta-3.jar:]
          at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36) [resteasy-jaxrs-3.0-beta-3.jar:]
          at io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:145) [undertow-servlet-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.servlet.core.ManagedServlet.getServlet(ManagedServlet.java:97) [undertow-servlet-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:87) [undertow-servlet-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:39) [undertow-servlet-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.server.HttpHandlers.executeHandler(HttpHandlers.java:46) [undertow-core-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:52) [undertow-core-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.server.HttpHandlers.executeHandler(HttpHandlers.java:46) [undertow-core-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.server.HttpHandlers.executeHandler(HttpHandlers.java:46) [undertow-core-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:66) [undertow-servlet-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.server.HttpHandlers.executeHandler(HttpHandlers.java:46) [undertow-core-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:51) [undertow-core-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.server.HttpHandlers.executeHandler(HttpHandlers.java:46) [undertow-core-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:71) [undertow-core-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.server.HttpHandlers.executeHandler(HttpHandlers.java:46) [undertow-core-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:26) [undertow-core-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:119) [undertow-servlet-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:107) [undertow-servlet-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.servlet.handlers.ServletInitialHandler.handleRequest(ServletInitialHandler.java:82) [undertow-servlet-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.server.HttpHandlers.executeRootHandler(HttpHandlers.java:53) [undertow-core-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:501) [undertow-core-1.0.0.Alpha6.jar:1.0.0.Alpha6]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09]
          at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09]

[jaikiran]

What is the OS and Java setup where this is failing? The reason I ask is because, we have the smoke tests passing fine against various CI jobs and even locally.

[jaikiran]

By the way, what is the thread stack size on the system where this is reproducible? It might just be that the stack size required by the threads might have increased recently with the introduction of undertow and the handlers perhaps. But that's just a guess.

[paul.robinson]

Jaikiran,

Thanks for taking a look.

This is the command I run to reproduce:

./build.sh -f ./testsuite/integration/xts/pom.xml -Pxts.integration.tests.profile clean test

Fails on:

jaime, CentOS 5.8 32bit, JDK 1.7.0_09

tenafly, CentOS 6 32bit, JDK 1.7.0_17

Passes on:

beacon, CentOS 5.6, 64bit, JDK 1.7.0_17

hoboken, CentOS 6, 64bit, JDK 1.7.0_17

Seems to only fail on 32bit machines: jaime & tenafly. Succeeds on 64 bit beacon and hoboken.

jaime is running JDK 1.7.0_09, where as hoboken runs jdk1.7.0_17. I tried running the test many times (~5) on hoboken with JDK 1.7.0_09, but I couldn't get it to fail.

As it was only failing on 32bit machines I tried running with -server to more closely match the 64bit vms. This passes. So it seems like we need some of the increased resources provisioned for -server. I checked the ThreadStackSize in particular and it's set by default to 320 for both -client and -server.

For completeness I tried using "java -client" on a 64bit vm (hoboken), but this didn't cause the test to fail. So it seems like it only fails on 32bit java vms running in client mode.

Is there a requirement that you have to run java in server mode to run the AS integration tests?

Paul.

[ctomc]

That is interesting problem...

just for completnes of problem, ligthning runs all as8 builds on 32bit jdk...

Also we ware testing 7.2 build on 32bit jdk reguraly (windows run was always 32bit)

so i doubt it is an issue with 32bit jvm, but i can only say that for AS codebase, if your run some additional tests out of our scope, it might be possible

[jaikiran]

If you take a look at our startup scripts (standalone.sh for example) we have a certain logic in there which sets the -server option while starting the server. The relevant piece is here:

if [ "x$CLIENT_SET" != "x" ]; then
        CLIENT_VM=true
    elif [ "x$SERVER_SET" = "x" ]; then
        if $darwin && [ "$JVM_OPTVERSION" = "-d32" ]; then
            # Prefer client for Macs, since they are primarily used for development
            CLIENT_VM=true
            PREPEND_JAVA_OPTS="$PREPEND_JAVA_OPTS -client"
        else
            PREPEND_JAVA_OPTS="$PREPEND_JAVA_OPTS -server"
        fi
    fi

That's via the scripts. Now since the tests are triggered via Arquillian which starts up the container, I'm guessing that it's not passing along the -server to the startup options. Having said that, I'm not sure why you are seeing this now.

To narrow this down, would you be willing to switch back to a revision of upstream, after which you believe you started seeing this problem? If running the tests against the 32 bit system passes on that revision then we might be able to understand this better.

[jaikiran]

Tomaz Cerar wrote:

so i doubt it is an issue with 32bit jvm, but i can only say that for AS codebase, if your run some additional tests out of our scope, it might be possible

From what Paul posted, he's actually running our testsuite, the integration/xts module in particular. I believe lightning runs it too. Can you check what (default) thread stack size lightning is on for these runs?

[ctomc]

jaikiran pai wrote:

From what Paul posted, he's actually running our testsuite, the integration/xts module in particular. I believe lightning runs it too. Can you check what (default) thread stack size lightning is on for these runs?

sure...

this is for pull requests on lightning: (bit bigger than actually needed just to make sure PRs are build successfuly)

MAVEN_OPTS="-Xmx1024m -Xms512m -XX:MaxPermSize=512m -XX:PermSize=256m"

on my TeamCity I use:

-Xms256m -Xmx512m -XX:MaxPermSize=256M

and also there are no issues...

--

tomaz

[paul.robinson]

Thanks Jaikiran and Tomaz,

I'm not sure setting MAVEN_OPTS would work as Arquillian is starting the server with params set in "testsuite/pom.xml". I tried increasing the memory here by changing the property to:

<surefire.memory.args>-Xmx1024m -Xms512m -XX:MaxPermSize=512m -XX:PermSize=256m</surefire.memory.args>

This seemed to be passed through to the server that Arquillian starts:

11:56:06,491 INFO  [org.jboss.as.arquillian.container.managed.ManagedDeployableContainer] (main) Starting container with: [/usr/local/jdk1.7.0_09/bin/java, -Xmx1024m, -Xms512m, -XX:MaxPermSize=512m, -XX:PermSize=256m, -Djboss.dist=/home/h
udson/workspace/narayana-JBTM-1622/./testsuite/integration/xts/../../../build/target/jboss-as-8.0.0.Alpha1-SNAPSHOT, -Djava.net.preferIPv4Stack=true, -Djava.net.preferIPv6Addresses=false, -Dts.timeout.factor=100, -Dnode0=127.0.0.1, -Dnode
1=127.0.0.1, -Dmcast=230.0.0.4, -Djbossas.ts.submodule.dir=/home/hudson/workspace/narayana-JBTM-1622/./testsuite/integration/xts, -Djbossas.ts.integ.dir=/home/hudson/workspace/narayana-JBTM-1622/./testsuite/integration/xts/.., -Djbossas.t
s.dir=/home/hudson/workspace/narayana-JBTM-1622/./testsuite/integration/xts/../.., -Djbossas.project.dir=/home/hudson/workspace/narayana-JBTM-1622/./testsuite/integration/xts/../../.., -Djboss.dist=/home/hudson/workspace/narayana-JBTM-162
2/./testsuite/integration/xts/../../../build/target/jboss-as-8.0.0.Alpha1-SNAPSHOT, -Djboss.inst=/home/hudson/workspace/narayana-JBTM-1622/./testsuite/integration/xts/target/jbossas, -Dorg.jboss.byteman.verbose, -Djboss.modules.system.pkg
s=org.jboss.byteman, -Dorg.jboss.byteman.transform.all, -javaagent:/home/hudson/workspace/narayana-JBTM-1622/./testsuite/integration/xts/target/lib/byteman.jar=listener:true, -ea, -Djboss.home.dir=/home/hudson/workspace/narayana-JBTM-1622
/./testsuite/integration/xts/target/jbossas, -Dorg.jboss.boot.log.file=/home/hudson/workspace/narayana-JBTM-1622/./testsuite/integration/xts/target/jbossas/standalone/log/boot.log, -Dlogging.configuration=file:/home/hudson/workspace/naray
ana-JBTM-1622/./testsuite/integration/xts/target/jbossas/standalone/configuration/logging.properties, -Djboss.bundles.dir=/home/hudson/workspace/narayana-JBTM-1622/./testsuite/integration/xts/../../../build/target/jboss-as-8.0.0.Alpha1-SN
APSHOT/bundles, -jar, /home/hudson/workspace/narayana-JBTM-1622/./testsuite/integration/xts/target/jbossas/jboss-modules.jar, -mp, /home/hudson/workspace/narayana-JBTM-1622/./testsuite/integration/xts/../../../build/target/jboss-as-8.0.0.
Alpha1-SNAPSHOT/modules:/home/hudson/workspace/narayana-JBTM-1622/./testsuite/integration/xts/target/modules, org.jboss.as.standalone, -server-config, standalone-xts.xml]

But I still see the StackOverflowError :-(

FYI, setting the following in "testsuite/pom.xml" fixes the problem, but it's probably not the best solution:

<jvm.args.other>-server</jvm.args.other>

I'll try a binary search to pinpoint the commit that introduced this issue. Shouldn't take too long. I think i can test one commit every 5mins.

Paul.

[jaikiran]

By the way, the right fix would be to start the server with -server option, like you did.

[paul.robinson]

jaikiran pai wrote:

By the way, the right fix would be to start the server with -server option, like you did.

Shall I raise a Jira and PR for this?

Would you still like me to find the exact commit that caused the issue?

[jaikiran]

Paul Robinson wrote:

jaikiran pai wrote:

By the way, the right fix would be to start the server with -server option, like you did.

Shall I raise a Jira and PR for this?

Yes please.

Paul Robinson wrote:

Would you still like me to find the exact commit that caused the issue?

Not really - I was (and still am) just curious why you started seeing this now. But it isn't really important to be spending too much time on, since it's just a matter of starting the server with the right options.

[sfcoy]

What does

{code:bash}$ ulimit -s{code}

show for the user accounts running the tests?

I get 10240 (x 1kB) on 32 bit CentOS 6.3, which is 10MB.

[paul.robinson]

Stephen,

I get 10240 too.

Paul.