EAP 6.4.0.GA start error: java.lang.ClassNotFoundException from Service Module Loader

[leo.chen]

Hi all,

I am deploying my project to JBoss EAP 6.4.0.GA version. The project was running fine on EAP 6.1.0 alpha version, but after moving to version 6.4.0.GA, a quick error was reported during startup. Below is the error information for your reference.

Can anybody see what the problem could be? Why just this servlet class (com.test.webservices.AemCXFNoSpringServlet) has this error while other servlet donot report such error? And also what do the ValidatorFactory missing and Validator missing mean? Let me know if you need anything detailed else to help analyze this problem. Thanks.

"{

    \"JBAS014671: Failed services\" => {\"jboss.deployment.subunit.\\\"test.ear\\\".\\\"test-web.war\\\".POST_MODULE\" => \"org.jboss.msc.service.StartException in service jboss.deployment.subunit.\\\"test.ear\\\".\\\"test-web.war\\\".POST_MODULE: JBAS018733: Failed to process phase POST_MODULE of subdeployment \\\"test-web.war\\\" of deployment \\\"test.ear\\\"

    Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: java.lang.ClassNotFoundException: com.test.webservices.AemCXFNoSpringServlet from [Module \\\"deployment.test.ear.test-web.war:main\\\" from Service Module Loader]

    Caused by: java.lang.ClassNotFoundException: com.test.webservices.AemCXFNoSpringServlet from [Module \\\"deployment.test.ear.test-web.war:main\\\" from Service Module Loader]\"},

    \"JBAS014771: Services with missing/unavailable dependencies\" => [

        \"jboss.naming.context.java.comp.test.test.TestMDB.ValidatorFactory missing [jboss.naming.context.java.comp.test.test.TestMDB]\",

        \"jboss.naming.context.java.comp.test.test.Test2MDB.Validator missing [jboss.naming.context.java.comp.test.test.Test2MDB]\"

    ]

}"

[leo.chen]

Here is the structure of the test.ear. Everything is the same when running with EAP 6.1.0.alpha version and there was no problem, but now I am having ClassNotFoundException for class com.test.webservices.AemCXFNoSpringServlet.

test.ear
   - lib
        - third party jars
        - classes folder
             - class (my project's class) folders (empty folders, without class files) and properties, resources files
   - META-INF
        - jboss-deployment-structure.xml
   - test-web.war
        - WEB-INF
             - jboss-web.xml
             - web.xml
        - jsp files, resource files
   - test.jar (with all my project's class files in it, including com.test.webservices.AemCXFNoSpringServlet class)

Content of jboss-web.xml:

<jboss-web>
   <context-root>test</context-root>
</jboss-web>

Content of jboss-deployment-structure.xml

<jboss-deployment-structure>
<ear-subdeployments-isolated>true</ear-subdeployments-isolated>
  <deployment>
      <dependencies>
        <module name="com.sun.xml.bind" export="true"/>
        <module name="javax.mail.api" export="true"/>
        <module name="org.apache.cxf" export="true"/>
        <module name="org.apache.cxf.impl" export="true"/>
      </dependencies>
  </deployment>
</jboss-deployment-structure>

One more note is the com.test.webservices.AemCXFNoSpringServlet class extends CXFNonSpringServlet class which is from modules\system\layers\base\org\apache\cxf\impl\main\cxf-rt-transports-http-2.7.14.redhat-1.jar. But since org.apache.cxf.impl module has been included in jboss-deployment-structure.xml (<module name="org.apache.cxf.impl" export="true"/>), I am blocked here with no more foundings. Please advise. Much appreciated. Let me know if any more information is wanted for analyzing.

[leo.chen]

Hi guys, any update for this thread?

[jaikiran]

Please post the entire exception stacktrace.

[lylewang]

ClassNotFoundException: com.test.webservices.AemCXFNoSpringServlet from [Module \\\"deployment.test.ear.test-web.war:main\\\" from Service Module Loader]

So the class "com.test.webservices.AemCXFNoSpringServlet" is packaged inside "test.ear/test.jar",

and your web app. "test-web.war" is trying to access the class "com.test.webservices.AemCXFNoSpringServlet" right ?

If I am understanding your story correctly, it might be related to the "subdeployment-isolated" you configured:

<ear-subdeployments-isolated>true</ear-subdeployments-isolated>

Check here: https://docs.jboss.org/author/display/AS72/Class+Loading+in+AS7

  <!-- Make sub deployments isolated by default, so they cannot see each others classes without a Class-Path entry -->

  <ear-subdeployments-isolated>true</ear-subdeployments-isolated>

Try to setup dependencies between sub-deployments, e.g.:

<sub-deploymentname="myapp.war">

    <!-- This corresponds to the module for a web deployment -->

    <!-- it can use all the same tags as the <deployment> entry above -->

    <dependencies>

      <!-- Adds a dependency on a ejb jar. This could also be done with a Class-Path entry -->

      <module name="deployment.myear.ear.myejbjar.jar" />

    </dependencies>

If this doesn't help, probably need a reproducer project so that I can have a play on my side.

[leo.chen]

Hi Jaikiran,

Thanks for your reply.

Here is the entire piece of log related to the errors:

2015-04-27 14:38:31,089;[MSC service thread 1-2];ERROR;org.jboss.msc.service.fail;MSC000001: Failed to start service jboss.deployment.subunit."test.ear"."test-web.war".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.subunit."test.ear"."test-web.war".POST_MODULE: JBAS018733: Failed to process phase  POST_MODULE of subdeployment "test-web.war" of deployment "test.ear"
 at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:166)
 at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980)
 at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 at java.lang.Thread.run(Thread.java:619)
Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: java.lang.ClassNotFoundException: com.test.webservices.AemCXFNoSpringServlet from [Module "deployment.test.ear.test-web.war:main" from Service Module Loader]
 at org.jboss.as.jaxrs.deployment.JaxrsScanningProcessor.checkDeclaredApplicationClassAsServlet(JaxrsScanningProcessor.java:287)
 at org.jboss.as.jaxrs.deployment.JaxrsScanningProcessor.scanWebDeployment(JaxrsScanningProcessor.java:152)
 at org.jboss.as.jaxrs.deployment.JaxrsScanningProcessor.deploy(JaxrsScanningProcessor.java:103)
 at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:159)
 ... 5 more
Caused by: java.lang.ClassNotFoundException: com.test.webservices.AemCXFNoSpringServlet from [Module "deployment.test.ear.test-web.war:main" from Service Module Loader]
 at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:213)
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459)
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:447)
 at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389)
 at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134)
 at org.jboss.as.jaxrs.deployment.JaxrsScanningProcessor.checkDeclaredApplicationClassAsServlet(JaxrsScanningProcessor.java:285)
 ... 8 more
2015-04-27 14:38:31,336;[Controller Boot Thread];ERROR;org.jboss.as.controller.management-operation;JBAS014612: Failed to operate（"deploy"） -- address：（[("deployment" => "test.ear")]）-- Failed operation：{
    "JBAS014671: Failed services" => {"jboss.deployment.subunit.\"test.ear\".\"test-web.war\".POST_MODULE" => "org.jboss.msc.service.StartException in service jboss.deployment.subunit.\"test.ear\".\"test-web.war\".POST_MODULE: JBAS018733: Failed to process phase  POST_MODULE of subdeployment \"test-web.war\" of deployment \"test.ear\"
    Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: java.lang.ClassNotFoundException: com.test.webservices.AemCXFNoSpringServlet from [Module \"deployment.test.ear.test-web.war:main\" from Service Module Loader]
    Caused by: java.lang.ClassNotFoundException: com.test.webservices.AemCXFNoSpringServlet from [Module \"deployment.test.ear.test-web.war:main\" from Service Module Loader]"},
    "JBAS014771: Services with missing/unavailable dependencies" => [
        "jboss.naming.context.java.comp.test.test.TestMDB.ValidatorFactory missing [jboss.naming.context.java.comp.test.test.TestMDB]",
        "jboss.naming.context.java.comp.test.test.Test2MDB.Validator missing [jboss.naming.context.java.comp.test.test.Test2MDB]"
    ]
}
2015-04-27 14:38:31,706;[ServerService Thread Pool -- 28];INFO ;org.jboss.as.server;JBAS015859: deployed "test.ear"（runtime-name: "test.ear"）
2015-04-27 14:38:31,716;[Controller Boot Thread];INFO ;org.jboss.as.controller;JBAS014774: Service status report
JBAS014775: New missing/unavailable dependencies：
service jboss.naming.context.java.comp.test.test.TestMDB (missing)dependencies：: [service jboss.naming.context.java.comp.test.test.TestMDB.ValidatorFactory, service jboss.naming.context.java.comp.test.test.TestMDB.Validator,] 
service jboss.naming.context.java.comp.test.test.Test2MDB (missing)dependencies: [service jboss.naming.context.java.comp.test.test.Test2MDB.ValidatorFactory, service jboss.naming.context.java.comp.test.test.Test2MDB.Validator] 
JBAS014777: Failed services：      service jboss.deployment.subunit."test.ear"."test-web.war".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.subunit."test.ear"."test-web.war".POST_MODULE: JBAS018733: Failed to process phase  POST_MODULE of subdeployment "test-web.war" of deployment "test.ear" 

[leo.chen]

Hi Lyle,

Thanks for your help. It look right and I have tried your suggestion, but it did not startup successfully with another kind of error messages.

I can confirm the  com.test.cxe.servlet.cms.teamsite.TranslateServlet is in test.jar and org.apache.regexp.RESyntaxException is in a third party jar under test2.war/WEB-INF\lib (test2.war is at the same directory as test-web.war).

Got any idea for this kind of error? What surprises me is everything was working fine with EAP 6.1.0.alpha version without any change on the class loading configuration. But errrs happen after moving to EAP 6.4.0.GA.

Here is the log trace:

2015-04-27 15:03:34,067;[MSC service thread 1-4];ERROR;org.jboss.msc.service.fail;MSC000001: Failed to start service jboss.deployment.subunit."test.ear"."test-web.war".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.subunit."test.ear"."test-web.war".POST_MODULE: JBAS018733: Failed to process phase of subdeployment "test-web.war" of deployment "test.ear"
 at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:166)
 at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980)
 at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: JBAS018757: Failed to get reflect information of class com.test.cxe.servlet.cms.teamsite.TranslateServlet of ModuleClassLoader for Module "deployment.test.ear.test.jar:main" from Service Module Loader 
 at org.jboss.as.server.deployment.reflect.DeploymentReflectionIndex.getClassIndex(DeploymentReflectionIndex.java:72)
 at org.jboss.as.ee.metadata.MethodAnnotationAggregator.runtimeAnnotationInformation(MethodAnnotationAggregator.java:58)
 at org.jboss.as.ee.component.deployers.InterceptorAnnotationProcessor.handleAnnotations(InterceptorAnnotationProcessor.java:107)
 at org.jboss.as.ee.component.deployers.InterceptorAnnotationProcessor.processComponentConfig(InterceptorAnnotationProcessor.java:92)
 at org.jboss.as.ee.component.deployers.InterceptorAnnotationProcessor.deploy(InterceptorAnnotationProcessor.java:77)
 at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:159)
 ... 5 more
Caused by: java.lang.NoClassDefFoundError: org/apache/regexp/RESyntaxException
 at java.lang.Class.getDeclaredFields0(Native Method)
 at java.lang.Class.privateGetDeclaredFields(Class.java:2291)
 at java.lang.Class.getDeclaredFields(Class.java:1743)
 at org.jboss.as.server.deployment.reflect.ClassReflectionIndex.<init>(ClassReflectionIndex.java:57)
 at org.jboss.as.server.deployment.reflect.DeploymentReflectionIndex.getClassIndex(DeploymentReflectionIndex.java:68)
 ... 10 more
Caused by: java.lang.ClassNotFoundException: org.apache.regexp.RESyntaxException from [Module "deployment.test.ear.test.jar:main" from Service Module Loader]
 at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:213)
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459)
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:447)
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:414)
 at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:414)
 at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389)
 at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134)
 at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
 ... 15 more

2015-04-27 15:03:34,347;[Controller Boot Thread];ERROR;org.jboss.as.controller.management-operation;JBAS014612: Failed to operate（"deploy"） -- address：（[("deployment" => "test.ear")]）-- Failed operation：{
    "JBAS014671: Failed services" => {"jboss.deployment.subunit.\"test.ear\".\"test-web.war\".POST_MODULE" => "org.jboss.msc.service.StartException in service jboss.deployment.subunit.\"test.ear\".\"test-web.war\".POST_MODULE: JBAS018733: Failed to process phase of subdeployment \"test-web.war\" of deployment \"test.ear\" 
    Caused by: java.lang.RuntimeException: JBAS018757: Failed to get reflect information of class com.test.cxe.servlet.cms.teamsite.TranslateServlet of ModuleClassLoader for Module \"deployment.test.ear.test.jar:main\" from Service Module Loader 
    Caused by: java.lang.NoClassDefFoundError: org/apache/regexp/RESyntaxException
    Caused by: java.lang.ClassNotFoundException: org.apache.regexp.RESyntaxException from [Module \"deployment.test.ear.test.jar:main\" from Service Module Loader]"},
    "JBAS014771: Services with missing/unavailable dependencies" => [
        "jboss.naming.context.java.comp.test.test.TestMDB.ValidatorFactory missing [jboss.naming.context.java.comp.test.test.TestMDB]",
        "jboss.naming.context.java.comp.test.test.Test2MDB.Validator missing [jboss.naming.context.java.comp.test.test.Test2MDB]"
    ]
}
2015-04-27 14:38:31,706;[ServerService Thread Pool -- 28];INFO ;org.jboss.as.server;JBAS015859: deployed "test.ear"（runtime-name: "test.ear"）
2015-04-27 14:38:31,716;[Controller Boot Thread];INFO ;org.jboss.as.controller;JBAS014774: Service status report
JBAS014775: New missing/unavailable dependencies：
service jboss.naming.context.java.comp.test.test.TestMDB (missing)dependencies：: [service jboss.naming.context.java.comp.test.test.TestMDB.ValidatorFactory, service jboss.naming.context.java.comp.test.test.TestMDB.Validator,] 
service jboss.naming.context.java.comp.test.test.Test2MDB (missing)dependencies: [service jboss.naming.context.java.comp.test.test.Test2MDB.ValidatorFactory, service jboss.naming.context.java.comp.test.test.Test2MDB.Validator] 
JBAS014777: Failed services：      service jboss.deployment.subunit."test.ear"."test-web.war".POST_MODULE: org.jboss.msc.service.StartException in service jboss.deployment.subunit."test.ear"."test-web.war".POST_MODULE: JBAS018733: Failed to process phase  POST_MODULE of subdeployment "test-web.war" of deployment "test.ear"

[lylewang]

If "org.apache.regexp" package is used by multiple subdeployments and it's a 3rd-party utility (looks like an apache tool), then probably better configure it as a custom global module.

Example here: http://www.mastertheboss.com/jboss-server/jboss-as-7/how-to-install-a-module-on-jboss-as-7

And then reference this global module dependency in your ear's jboss-deployment-structure.xml

[leo.chen]

Hi Lyle,

I changed the reference to com/sun/org/apache/regexp/internal in java library rt.jar for the RE.class and added to jboss-deployment-structure.xml

<system export="true">               
    <paths>                   
    <path name="com/sun/org/apache/regexp/internal"/>               
    </paths>           
</system>

and this resolves current errors now. Thanks so much for your help on this. Really appreciated.

One thing is for based on first reply, I tried to set false of <ear-subdeployments-isolated>false</ear-subdeployments-isolated> intead of adding the sub-deployment, and it also works. So do you have any advise on this? What is the difference between this two changes? Is it ok to just set the isolated attribute to false? Any potential issues if using false?

[lylewang]

Your questions are answered in the article already : https://docs.jboss.org/author/display/AS72/Class+Loading+in+AS7

Read through the "EAR Class Loading" section please