1 2 Previous Next 23 Replies Latest reply on Jul 26, 2017 3:14 AM by jiseb

Random NoClassDefFoundError in WildFly

jiseb Jun 20, 2017 7:47 AM

Hello,

We are struggeling with one issue for many weeks and we are running out of ideas.

We are have a set of ear modules (with war and ejb nested deployments) deployed in WildFly 10.1.0. Each module has some servlet and handles requests to the servlet (some transaction processing). Subset of these modules work fine in every situation. But small subset of modules is sometimes encountering NoClassDefFoundError error on some classes (while processing the request). These classes are definetely in the module and have no static constructor. No other error can be seen in any logs. Then we just do unassing/assign for this module and it starts to work correcly. But later (usually after deployment of new version or after restart of WF) the problem is back (just the class causing this error may change).

I will be happy for any idea that will help us to solve this. Thank you in advance,

Jiri

1. Re: Random NoClassDefFoundError in WildFly

ctomc Jun 20, 2017 8:23 AM (in response to jiseb)

do you have some static cache somewhere?
Actions
2. Re: Random NoClassDefFoundError in WildFly

jiseb Jun 20, 2017 8:31 AM (in response to ctomc)

No, there is no static cache. I fact we have removed all static fields from these classes so we are quite sure there is no issue during static initialization. We have seen this problem also with an enum.
Actions
3. Re: Random NoClassDefFoundError in WildFly

ctomc Jun 20, 2017 8:39 AM (in response to jiseb)

do you maybe have some duplicate classes (jars) somwhere in your deployment, that could also cause something like that.

could you paste structure of your deployment(s), mostly the list of jars and where they are.
usually you can get that list with jar tf <name-of-ear-or-war>

also if you have any jboss-deployment-structure.xml files present, please post them.
Actions
4. Re: Random NoClassDefFoundError in WildFly

jiseb Jun 20, 2017 9:37 AM (in response to ctomc)

We have alraedy checked for duplicates but were not able to see any.

Here is the ear structure (I can also provide content of jar and war submodules if needed):
META-INF/
META-INF/MANIFEST.MF
lib/
EpascustomerHostConnector-war-1.11.10.war
EpascustomerHostConnector-ejb-1.11.10.jar
META-INF/jboss-deployment-structure.xml
META-INF/application.xml
META-INF/maven/
META-INF/maven/cz.company.customer.loom.host.epas/
META-INF/maven/cz.company.customer.loom.host.epas/EpascustomerHostConnector-ear/
META-INF/maven/cz.company.customer.loom.host.epas/EpascustomerHostConnector-ear/pom.xml
META-INF/maven/cz.company.customer.loom.host.epas/EpascustomerHostConnector-ear/pom.properties

Here is the content of jboss-deployment-structure.xml:
<jboss-deployment-structure>

<ear-subdeployments-isolated>false</ear-subdeployments-isolated>
</jboss-deployment-structure>

I believe all classes encountering this issue are from war.
Actions
5. Re: Random NoClassDefFoundError in WildFly

ctomc Jun 20, 2017 10:28 AM (in response to jiseb)

can you provide the same for all wars,
mostly the list of jars in WEB-INF/lib

and list of jars in lib of ear, which is missing above.
Actions
6. Re: Random NoClassDefFoundError in WildFly

jiseb Jun 20, 2017 11:33 AM (in response to ctomc)

Here is the complete structure (most common problems are with class TransactionContextImpl and enum FieldRequestIccAllowedTags):
EAR structure:

META-INF/
META-INF/MANIFEST.MF
lib/
lib/ISO8583-1.26.jar
lib/ProtoX-2.84.jar
lib/CodePool-1.88.jar
lib/SHSM-1.82.jar
lib/LoomBase-2.30.7.jar
EpascustomerHostConnector-war-1.11.10.war
EpascustomerHostConnector-ejb-1.11.10.jar
META-INF/jboss-deployment-structure.xml
META-INF/application.xml
META-INF/maven/

EpascustomerHostConnector-ejb-1.11.10.jar structure:

META-INF/
META-INF/MANIFEST.MF
xjb/
xsd/
cz/
cz/company/
....
META-INF/ejb-jar.xml

EpascustomerHostConnector-war-1.11.10.war structure

META-INF/
META-INF/MANIFEST.MF
WEB-INF/
WEB-INF/classes/
WEB-INF/classes/cz/
WEB-INF/classes/cz/company/
WEB-INF/lib/
index.jsp
WEB-INF/classes/hibernate.cfg.xml
WEB-INF/lib/ISO8583-1.26.jar
WEB-INF/lib/opencsv-1.8.jar
WEB-INF/glassfish-web.xml
WEB-INF/sun-web.xml.old
WEB-INF/web.xml
WEB-INF/glassfish-resources.xml
Actions
7. Re: Random NoClassDefFoundError in WildFly

ctomc Jun 20, 2017 11:35 AM (in response to jiseb)

you have ISO8583-1.26.jar in EAR/lib and WAR/WEB-INF/lib
which could cause issues.

are the classes you get CNFE present in this jar?

also, does changing
<ear-subdeployments-isolated>false</ear-subdeployments-isolated>
to true, change anything?
Actions
8. Re: Random NoClassDefFoundError in WildFly

jiseb Jun 20, 2017 12:40 PM (in response to ctomc)

We will try to remove ISO8583-1.26.jar but I am afraid it will not help. All problematic classes are directly from war archive (not from libraries).
If we change ear-subdeployments-isolated to true then Im afraid classes from war will not see classes from jar. Am I right?
Actions
9. Re: Random NoClassDefFoundError in WildFly

jaikiran Jun 21, 2017 12:10 AM (in response to jiseb)

Could you please paste the entire exception stacktrace when this happens? Typically stacktraces (along with relevant logs) give enough hints to suggest what's going on.
Actions
10. Re: Random NoClassDefFoundError in WildFly

jiseb Jun 21, 2017 12:44 AM (in response to jaikiran)

I will paste two stacktraces as it is different for the first and all other occurences. In the first occurance it is caused by ClassNotFoundException.
First occurence:

2017-06-20 11:31:07,262 ERROR [io.undertow.request] (default task-97) UT005023: Exception handling request to /EpasHostConnector/xproto: java.lang.NoClassDefFoundError: cz/company/loom/host/epas/context/TransactionContextImpl
    at cz.company.loom.host.epas.XProtoHostServlet.handleDocument(XProtoHostServlet.java:138)
    at cz.company.loom.data.TransactionHandlerServletBase.doPost(TransactionHandlerServletBase.java:44)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
    at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
    at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: cz.company.loom.host.epas.context.TransactionContextImpl from [Module "deployment.EpascustomerHostConnector-ear-1.11.9.1-host.ear.EpascustomerHostConnector-war-1.11.9.1.war:main" from Service Module Loader]
    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351)
    at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93)
    ... 31 more

All following occurences:

2017-06-20 11:31:08,574 ERROR [io.undertow.request] (default task-181) UT005023: Exception handling request to /EpasHostConnector/host/xproto: java.lang.NoClassDefFoundError: cz/company/loom/host/epas/context/TransactionContextImpl
    at cz.company.loom.host.epas.XProtoHostServlet.handleDocument(XProtoHostServlet.java:138)
    at cz.company.loom.data.TransactionHandlerServletBase.doPost(TransactionHandlerServletBase.java:44)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
    at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
    at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Actions
11. Re: Random NoClassDefFoundError in WildFly

jaikiran Jun 21, 2017 12:49 AM (in response to jiseb)

Caused by: java.lang.ClassNotFoundException: cz.company.loom.host.epas.context.TransactionContextImpl from [Module "deployment.EpascustomerHostConnector-ear-1.11.9.1-host.ear.EpascustomerHostConnector-war-1.11.9.1.war:main" from Service Module Loader]

Looking at the war contents that you posted in one of your replies, the cz.company.loom.host.epas.context.TransactionContextImpl class isn't present in your .war file. Where exactly is that class located?
Actions
12. Re: Random NoClassDefFoundError in WildFly

jiseb Jun 21, 2017 1:17 AM (in response to jaikiran)

It seems my previous comment was edited by somebody as it was very long. Here is the location of the class in the war (posting the whle package content):
WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextException.class
WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextBase.class
WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContext.class
WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextDiagnostic.class
WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextImpl.class
WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextDeclined.class
WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextFailed.class

I am pretty sure the class is in the package because the module works correctly after unassign+assing (and also before the first occurance).
Actions
13. Re: Random NoClassDefFoundError in WildFly

jaikiran Jun 21, 2017 1:28 AM (in response to jiseb)

Jiri Sebesta wrote:

I am pretty sure the class is in the package because the module works correctly after unassign+assing (and also before the first occurance).
When you say unassign + assign, what do you mean? Are you talking about redeploying the application?

Anyway, the best way to debug this is to enable TRACE level logging of org.jboss.modules package and see what's going on. It logs detailed messages about the module classloading that happens.
Actions
14. Re: Random NoClassDefFoundError in WildFly

jiseb Jun 21, 2017 1:50 AM (in response to jaikiran)

Yes, I mean redeploying the ear module. Thanks for the tip with trace logging, we wil try this.
Actions

1 2 Previous Next

Go to original post