Random NoClassDefFoundError in WildFly

[jiseb]

Hello,

We are struggeling with one issue for many weeks and we are running out of ideas.

We are have a set of ear modules (with war and ejb nested deployments) deployed in WildFly  10.1.0. Each module has some servlet and handles requests to the servlet (some transaction processing). Subset of these modules work fine in every situation. But small subset of modules is sometimes encountering NoClassDefFoundError error on some classes (while processing the request). These classes are definetely in the module and have no static constructor. No other error can be seen in any logs. Then we just do unassing/assign for this module and it starts to work correcly. But later (usually after deployment of new version or after restart of WF) the problem is back (just the class causing this error may change).

I will be happy for any idea that will help us to solve this. Thank you in advance,

Jiri

[ctomc]

do you have some static cache somewhere?

[jiseb]

No, there is no static cache. I fact we have removed all static fields from these classes so we are quite sure there is no issue during static initialization. We have seen this problem also with an enum.

[ctomc]

do you maybe have some duplicate classes (jars) somwhere in your deployment, that could also cause something like that.

could you paste structure of your deployment(s), mostly the list of jars and where they are.

usually you can get that list with jar tf <name-of-ear-or-war>

also if you have any jboss-deployment-structure.xml files present, please post them.

[jiseb]

We have alraedy checked for duplicates but were not able to see any.

Here is the ear structure (I can also provide content of jar and war submodules if needed):

META-INF/

META-INF/MANIFEST.MF

lib/

EpascustomerHostConnector-war-1.11.10.war

EpascustomerHostConnector-ejb-1.11.10.jar

META-INF/jboss-deployment-structure.xml

META-INF/application.xml

META-INF/maven/

META-INF/maven/cz.company.customer.loom.host.epas/

META-INF/maven/cz.company.customer.loom.host.epas/EpascustomerHostConnector-ear/

META-INF/maven/cz.company.customer.loom.host.epas/EpascustomerHostConnector-ear/pom.xml

META-INF/maven/cz.company.customer.loom.host.epas/EpascustomerHostConnector-ear/pom.properties

Here is the content of jboss-deployment-structure.xml:

<jboss-deployment-structure>

    <!-- Make sub deployments isolated by default, so they cannot see each others classes without a Class-Path entry -->

    <ear-subdeployments-isolated>false</ear-subdeployments-isolated>

</jboss-deployment-structure>

I believe all classes encountering this issue are from war.

[ctomc]

can you provide the same for all wars,

mostly the list of jars in WEB-INF/lib

and list of jars in lib of ear, which is missing above.

[jiseb]

Here is the complete structure (most common problems are with class TransactionContextImpl and enum FieldRequestIccAllowedTags):

EAR structure:

META-INF/

META-INF/MANIFEST.MF

lib/

lib/ISO8583-1.26.jar

lib/ProtoX-2.84.jar

lib/CodePool-1.88.jar

lib/SHSM-1.82.jar

lib/LoomBase-2.30.7.jar

EpascustomerHostConnector-war-1.11.10.war

EpascustomerHostConnector-ejb-1.11.10.jar

META-INF/jboss-deployment-structure.xml

META-INF/application.xml

META-INF/maven/

EpascustomerHostConnector-ejb-1.11.10.jar structure:

META-INF/

META-INF/MANIFEST.MF

xjb/

xsd/

cz/

cz/company/

....

META-INF/ejb-jar.xml

EpascustomerHostConnector-war-1.11.10.war structure

META-INF/

META-INF/MANIFEST.MF

WEB-INF/

WEB-INF/classes/

WEB-INF/classes/cz/

WEB-INF/classes/cz/company/

WEB-INF/lib/

index.jsp

WEB-INF/classes/hibernate.cfg.xml

WEB-INF/lib/ISO8583-1.26.jar

WEB-INF/lib/opencsv-1.8.jar

WEB-INF/glassfish-web.xml

WEB-INF/sun-web.xml.old

WEB-INF/web.xml

WEB-INF/glassfish-resources.xml

[ctomc]

you have ISO8583-1.26.jar in EAR/lib and WAR/WEB-INF/lib

which could cause issues.

are the classes you get CNFE present in this jar?

also, does changing

  <ear-subdeployments-isolated>false</ear-subdeployments-isolated>

to true, change anything?

[jiseb]

We will try to remove ISO8583-1.26.jar but I am afraid it will not help. All problematic classes are directly from war archive (not from libraries).

If we change ear-subdeployments-isolated to true then Im afraid classes from war will not see classes from jar. Am I right?

[jaikiran]

Could you please paste the entire exception stacktrace when this happens? Typically stacktraces (along with relevant logs) give enough hints to suggest what's going on.

[jiseb]

I will paste two stacktraces as it is different for the first and all other occurences. In the first occurance it is caused by ClassNotFoundException.

First occurence:

2017-06-20 11:31:07,262 ERROR [io.undertow.request] (default task-97) UT005023: Exception handling request to /EpasHostConnector/xproto: java.lang.NoClassDefFoundError: cz/company/loom/host/epas/context/TransactionContextImpl

    at cz.company.loom.host.epas.XProtoHostServlet.handleDocument(XProtoHostServlet.java:138)

    at cz.company.loom.data.TransactionHandlerServletBase.doPost(TransactionHandlerServletBase.java:44)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

    at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

    at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

    at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)

    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.ClassNotFoundException: cz.company.loom.host.epas.context.TransactionContextImpl from [Module "deployment.EpascustomerHostConnector-ear-1.11.9.1-host.ear.EpascustomerHostConnector-war-1.11.9.1.war:main" from Service Module Loader]

    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)

    at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363)

    at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351)

    at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93)

    ... 31 more

All following occurences:

2017-06-20 11:31:08,574 ERROR [io.undertow.request] (default task-181) UT005023: Exception handling request to /EpasHostConnector/host/xproto: java.lang.NoClassDefFoundError: cz/company/loom/host/epas/context/TransactionContextImpl

    at cz.company.loom.host.epas.XProtoHostServlet.handleDocument(XProtoHostServlet.java:138)

    at cz.company.loom.data.TransactionHandlerServletBase.doPost(TransactionHandlerServletBase.java:44)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

    at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

    at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

    at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)

    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    at java.lang.Thread.run(Thread.java:745)

[jaikiran]

Caused by: java.lang.ClassNotFoundException: cz.company.loom.host.epas.context.TransactionContextImpl from [Module "deployment.EpascustomerHostConnector-ear-1.11.9.1-host.ear.EpascustomerHostConnector-war-1.11.9.1.war:main" from Service Module Loader]

Looking at the war contents that you posted in one of your replies, the cz.company.loom.host.epas.context.TransactionContextImpl class isn't present in your .war file. Where exactly is that class located?

[jiseb]

It seems my previous comment was edited by somebody as it was very long. Here is the location of the class in the war (posting the whle package content):

WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextException.class

WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextBase.class

WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContext.class

WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextDiagnostic.class

WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextImpl.class

WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextDeclined.class

WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextFailed.class

I am pretty sure the class is in the package because the module works correctly after unassign+assing (and also before the first occurance).

[jaikiran]

Jiri Sebesta wrote:

I am pretty sure the class is in the package because the module works correctly after unassign+assing (and also before the first occurance).

When you say unassign + assign, what do you mean? Are you talking about redeploying the application?

Anyway, the best way to debug this is to enable TRACE level logging of org.jboss.modules package and see what's going on. It logs detailed messages about the module classloading that happens.

[jiseb]

Yes, I mean redeploying the ear module. Thanks for the tip with trace logging, we wil try this.