1 2 Previous Next 23 Replies Latest reply on Jul 26, 2017 3:14 AM by jiseb

    Random NoClassDefFoundError in WildFly

    jiseb

      Hello,

      We are struggeling with one issue for many weeks and we are running out of ideas.

       

      We are have a set of ear modules (with war and ejb nested deployments) deployed in WildFly  10.1.0. Each module has some servlet and handles requests to the servlet (some transaction processing). Subset of these modules work fine in every situation. But small subset of modules is sometimes encountering NoClassDefFoundError error on some classes (while processing the request). These classes are definetely in the module and have no static constructor. No other error can be seen in any logs. Then we just do unassing/assign for this module and it starts to work correcly. But later (usually after deployment of new version or after restart of WF) the problem is back (just the class causing this error may change).

       

      I will be happy for any idea that will help us to solve this. Thank you in advance,

      Jiri

        • 1. Re: Random NoClassDefFoundError in WildFly
          ctomc

          do you have some static cache somewhere?

          • 2. Re: Random NoClassDefFoundError in WildFly
            jiseb

            No, there is no static cache. I fact we have removed all static fields from these classes so we are quite sure there is no issue during static initialization. We have seen this problem also with an enum.

            • 3. Re: Random NoClassDefFoundError in WildFly
              ctomc

              do you maybe have some duplicate classes (jars) somwhere in your deployment, that could also cause something like that.

               

              could you paste structure of your deployment(s), mostly the list of jars and where they are.

              usually you can get that list with jar tf <name-of-ear-or-war>

               

              also if you have any jboss-deployment-structure.xml files present, please post them.

              • 4. Re: Random NoClassDefFoundError in WildFly
                jiseb

                We have alraedy checked for duplicates but were not able to see any.

                 

                Here is the ear structure (I can also provide content of jar and war submodules if needed):

                META-INF/

                META-INF/MANIFEST.MF

                lib/

                EpascustomerHostConnector-war-1.11.10.war

                EpascustomerHostConnector-ejb-1.11.10.jar

                META-INF/jboss-deployment-structure.xml

                META-INF/application.xml

                META-INF/maven/

                META-INF/maven/cz.company.customer.loom.host.epas/

                META-INF/maven/cz.company.customer.loom.host.epas/EpascustomerHostConnector-ear/

                META-INF/maven/cz.company.customer.loom.host.epas/EpascustomerHostConnector-ear/pom.xml

                META-INF/maven/cz.company.customer.loom.host.epas/EpascustomerHostConnector-ear/pom.properties

                 

                Here is the content of jboss-deployment-structure.xml:

                <jboss-deployment-structure>

                    <!-- Make sub deployments isolated by default, so they cannot see each others classes without a Class-Path entry -->

                    <ear-subdeployments-isolated>false</ear-subdeployments-isolated>

                </jboss-deployment-structure>

                 

                I believe all classes encountering this issue are from war.

                • 5. Re: Random NoClassDefFoundError in WildFly
                  ctomc

                  can you provide the same for all wars,

                  mostly the list of jars in WEB-INF/lib

                   

                  and list of jars in lib of ear, which is missing above.

                  • 6. Re: Random NoClassDefFoundError in WildFly
                    jiseb

                    Here is the complete structure (most common problems are with class TransactionContextImpl and enum FieldRequestIccAllowedTags):

                    EAR structure:

                     

                    META-INF/

                    META-INF/MANIFEST.MF

                    lib/

                    lib/ISO8583-1.26.jar

                    lib/ProtoX-2.84.jar

                    lib/CodePool-1.88.jar

                    lib/SHSM-1.82.jar

                    lib/LoomBase-2.30.7.jar

                    EpascustomerHostConnector-war-1.11.10.war

                    EpascustomerHostConnector-ejb-1.11.10.jar

                    META-INF/jboss-deployment-structure.xml

                    META-INF/application.xml

                    META-INF/maven/

                     

                    EpascustomerHostConnector-ejb-1.11.10.jar structure:

                     

                    META-INF/

                    META-INF/MANIFEST.MF

                    xjb/

                    xsd/

                    cz/

                    cz/company/

                    ....

                    META-INF/ejb-jar.xml

                     

                    EpascustomerHostConnector-war-1.11.10.war structure

                     

                    META-INF/

                    META-INF/MANIFEST.MF

                    WEB-INF/

                    WEB-INF/classes/

                    WEB-INF/classes/cz/

                    WEB-INF/classes/cz/company/

                    WEB-INF/lib/

                    index.jsp

                    WEB-INF/classes/hibernate.cfg.xml

                    WEB-INF/lib/ISO8583-1.26.jar

                    WEB-INF/lib/opencsv-1.8.jar

                    WEB-INF/glassfish-web.xml

                    WEB-INF/sun-web.xml.old

                    WEB-INF/web.xml

                    WEB-INF/glassfish-resources.xml

                    • 7. Re: Random NoClassDefFoundError in WildFly
                      ctomc

                      you have ISO8583-1.26.jar in EAR/lib and WAR/WEB-INF/lib

                      which could cause issues.

                       

                      are the classes you get CNFE present in this jar?

                       

                      also, does changing

                        <ear-subdeployments-isolated>false</ear-subdeployments-isolated>

                      to true, change anything?

                      • 8. Re: Random NoClassDefFoundError in WildFly
                        jiseb

                        We will try to remove ISO8583-1.26.jar but I am afraid it will not help. All problematic classes are directly from war archive (not from libraries).

                        If we change ear-subdeployments-isolated to true then Im afraid classes from war will not see classes from jar. Am I right?

                        • 9. Re: Random NoClassDefFoundError in WildFly
                          jaikiran

                          Could you please paste the entire exception stacktrace when this happens? Typically stacktraces (along with relevant logs) give enough hints to suggest what's going on.

                          • 10. Re: Random NoClassDefFoundError in WildFly
                            jiseb

                            I will paste two stacktraces as it is different for the first and all other occurences. In the first occurance it is caused by ClassNotFoundException.

                            First occurence:

                             

                            2017-06-20 11:31:07,262 ERROR [io.undertow.request] (default task-97) UT005023: Exception handling request to /EpasHostConnector/xproto: java.lang.NoClassDefFoundError: cz/company/loom/host/epas/context/TransactionContextImpl

                                at cz.company.loom.host.epas.XProtoHostServlet.handleDocument(XProtoHostServlet.java:138)

                                at cz.company.loom.data.TransactionHandlerServletBase.doPost(TransactionHandlerServletBase.java:44)

                                at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

                                at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

                                at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

                                at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

                                at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

                                at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

                                at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

                                at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

                                at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

                                at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

                                at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

                                at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)

                                at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

                                at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

                                at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

                                at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

                                at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

                                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                                at java.lang.Thread.run(Thread.java:745)

                            Caused by: java.lang.ClassNotFoundException: cz.company.loom.host.epas.context.TransactionContextImpl from [Module "deployment.EpascustomerHostConnector-ear-1.11.9.1-host.ear.EpascustomerHostConnector-war-1.11.9.1.war:main" from Service Module Loader]

                                at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)

                                at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363)

                                at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351)

                                at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93)

                                ... 31 more

                             

                            All following occurences:

                             

                            2017-06-20 11:31:08,574 ERROR [io.undertow.request] (default task-181) UT005023: Exception handling request to /EpasHostConnector/host/xproto: java.lang.NoClassDefFoundError: cz/company/loom/host/epas/context/TransactionContextImpl

                                at cz.company.loom.host.epas.XProtoHostServlet.handleDocument(XProtoHostServlet.java:138)

                                at cz.company.loom.data.TransactionHandlerServletBase.doPost(TransactionHandlerServletBase.java:44)

                                at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

                                at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

                                at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

                                at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

                                at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

                                at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

                                at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

                                at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

                                at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

                                at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

                                at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

                                at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)

                                at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

                                at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

                                at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

                                at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

                                at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

                                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                                at java.lang.Thread.run(Thread.java:745)

                            • 11. Re: Random NoClassDefFoundError in WildFly
                              jaikiran

                              Caused by: java.lang.ClassNotFoundException: cz.company.loom.host.epas.context.TransactionContextImpl from [Module "deployment.EpascustomerHostConnector-ear-1.11.9.1-host.ear.EpascustomerHostConnector-war-1.11.9.1.war:main" from Service Module Loader]

                               

                              Looking at the war contents that you posted in one of your replies, the cz.company.loom.host.epas.context.TransactionContextImpl class isn't present in your .war file. Where exactly is that class located?

                              • 12. Re: Random NoClassDefFoundError in WildFly
                                jiseb

                                It seems my previous comment was edited by somebody as it was very long. Here is the location of the class in the war (posting the whle package content):

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextException.class

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextBase.class

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContext.class

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextDiagnostic.class

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextImpl.class

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextDeclined.class

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextFailed.class

                                 

                                I am pretty sure the class is in the package because the module works correctly after unassign+assing (and also before the first occurance).

                                • 13. Re: Random NoClassDefFoundError in WildFly
                                  jaikiran

                                  Jiri Sebesta wrote:

                                   

                                   

                                  I am pretty sure the class is in the package because the module works correctly after unassign+assing (and also before the first occurance).

                                  When you say unassign + assign, what do you mean? Are you talking about redeploying the application?

                                   

                                  Anyway, the best way to debug this is to enable TRACE level logging of org.jboss.modules package and see what's going on. It logs detailed messages about the module classloading that happens.

                                  • 14. Re: Random NoClassDefFoundError in WildFly
                                    jiseb

                                    Yes, I mean redeploying the ear module. Thanks for the tip with trace logging, we wil try this.

                                    1 2 Previous Next