1 2 Previous Next 23 Replies Latest reply on Jul 26, 2017 3:14 AM by Jiri Sebesta

    Random NoClassDefFoundError in WildFly

    Jiri Sebesta Newbie

      Hello,

      We are struggeling with one issue for many weeks and we are running out of ideas.

       

      We are have a set of ear modules (with war and ejb nested deployments) deployed in WildFly  10.1.0. Each module has some servlet and handles requests to the servlet (some transaction processing). Subset of these modules work fine in every situation. But small subset of modules is sometimes encountering NoClassDefFoundError error on some classes (while processing the request). These classes are definetely in the module and have no static constructor. No other error can be seen in any logs. Then we just do unassing/assign for this module and it starts to work correcly. But later (usually after deployment of new version or after restart of WF) the problem is back (just the class causing this error may change).

       

      I will be happy for any idea that will help us to solve this. Thank you in advance,

      Jiri

        • 1. Re: Random NoClassDefFoundError in WildFly
          Tomaz Cerar Master

          do you have some static cache somewhere?

          • 2. Re: Random NoClassDefFoundError in WildFly
            Jiri Sebesta Newbie

            No, there is no static cache. I fact we have removed all static fields from these classes so we are quite sure there is no issue during static initialization. We have seen this problem also with an enum.

            • 3. Re: Random NoClassDefFoundError in WildFly
              Tomaz Cerar Master

              do you maybe have some duplicate classes (jars) somwhere in your deployment, that could also cause something like that.

               

              could you paste structure of your deployment(s), mostly the list of jars and where they are.

              usually you can get that list with jar tf <name-of-ear-or-war>

               

              also if you have any jboss-deployment-structure.xml files present, please post them.

              • 4. Re: Random NoClassDefFoundError in WildFly
                Jiri Sebesta Newbie

                We have alraedy checked for duplicates but were not able to see any.

                 

                Here is the ear structure (I can also provide content of jar and war submodules if needed):

                META-INF/

                META-INF/MANIFEST.MF

                lib/

                EpascustomerHostConnector-war-1.11.10.war

                EpascustomerHostConnector-ejb-1.11.10.jar

                META-INF/jboss-deployment-structure.xml

                META-INF/application.xml

                META-INF/maven/

                META-INF/maven/cz.company.customer.loom.host.epas/

                META-INF/maven/cz.company.customer.loom.host.epas/EpascustomerHostConnector-ear/

                META-INF/maven/cz.company.customer.loom.host.epas/EpascustomerHostConnector-ear/pom.xml

                META-INF/maven/cz.company.customer.loom.host.epas/EpascustomerHostConnector-ear/pom.properties

                 

                Here is the content of jboss-deployment-structure.xml:

                <jboss-deployment-structure>

                    <!-- Make sub deployments isolated by default, so they cannot see each others classes without a Class-Path entry -->

                    <ear-subdeployments-isolated>false</ear-subdeployments-isolated>

                </jboss-deployment-structure>

                 

                I believe all classes encountering this issue are from war.

                • 5. Re: Random NoClassDefFoundError in WildFly
                  Tomaz Cerar Master

                  can you provide the same for all wars,

                  mostly the list of jars in WEB-INF/lib

                   

                  and list of jars in lib of ear, which is missing above.

                  • 6. Re: Random NoClassDefFoundError in WildFly
                    Jiri Sebesta Newbie

                    Here is the complete structure (most common problems are with class TransactionContextImpl and enum FieldRequestIccAllowedTags):

                    EAR structure:

                     

                    META-INF/

                    META-INF/MANIFEST.MF

                    lib/

                    lib/ISO8583-1.26.jar

                    lib/ProtoX-2.84.jar

                    lib/CodePool-1.88.jar

                    lib/SHSM-1.82.jar

                    lib/LoomBase-2.30.7.jar

                    EpascustomerHostConnector-war-1.11.10.war

                    EpascustomerHostConnector-ejb-1.11.10.jar

                    META-INF/jboss-deployment-structure.xml

                    META-INF/application.xml

                    META-INF/maven/

                     

                    EpascustomerHostConnector-ejb-1.11.10.jar structure:

                     

                    META-INF/

                    META-INF/MANIFEST.MF

                    xjb/

                    xsd/

                    cz/

                    cz/company/

                    ....

                    META-INF/ejb-jar.xml

                     

                    EpascustomerHostConnector-war-1.11.10.war structure

                     

                    META-INF/

                    META-INF/MANIFEST.MF

                    WEB-INF/

                    WEB-INF/classes/

                    WEB-INF/classes/cz/

                    WEB-INF/classes/cz/company/

                    WEB-INF/lib/

                    index.jsp

                    WEB-INF/classes/hibernate.cfg.xml

                    WEB-INF/lib/ISO8583-1.26.jar

                    WEB-INF/lib/opencsv-1.8.jar

                    WEB-INF/glassfish-web.xml

                    WEB-INF/sun-web.xml.old

                    WEB-INF/web.xml

                    WEB-INF/glassfish-resources.xml

                    • 7. Re: Random NoClassDefFoundError in WildFly
                      Tomaz Cerar Master

                      you have ISO8583-1.26.jar in EAR/lib and WAR/WEB-INF/lib

                      which could cause issues.

                       

                      are the classes you get CNFE present in this jar?

                       

                      also, does changing

                        <ear-subdeployments-isolated>false</ear-subdeployments-isolated>

                      to true, change anything?

                      • 8. Re: Random NoClassDefFoundError in WildFly
                        Jiri Sebesta Newbie

                        We will try to remove ISO8583-1.26.jar but I am afraid it will not help. All problematic classes are directly from war archive (not from libraries).

                        If we change ear-subdeployments-isolated to true then Im afraid classes from war will not see classes from jar. Am I right?

                        • 9. Re: Random NoClassDefFoundError in WildFly
                          jaikiran pai Master

                          Could you please paste the entire exception stacktrace when this happens? Typically stacktraces (along with relevant logs) give enough hints to suggest what's going on.

                          • 10. Re: Random NoClassDefFoundError in WildFly
                            Jiri Sebesta Newbie

                            I will paste two stacktraces as it is different for the first and all other occurences. In the first occurance it is caused by ClassNotFoundException.

                            First occurence:

                             

                            2017-06-20 11:31:07,262 ERROR [io.undertow.request] (default task-97) UT005023: Exception handling request to /EpasHostConnector/xproto: java.lang.NoClassDefFoundError: cz/company/loom/host/epas/context/TransactionContextImpl

                                at cz.company.loom.host.epas.XProtoHostServlet.handleDocument(XProtoHostServlet.java:138)

                                at cz.company.loom.data.TransactionHandlerServletBase.doPost(TransactionHandlerServletBase.java:44)

                                at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

                                at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

                                at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

                                at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

                                at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

                                at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

                                at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

                                at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

                                at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

                                at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

                                at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

                                at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)

                                at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

                                at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

                                at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

                                at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

                                at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

                                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                                at java.lang.Thread.run(Thread.java:745)

                            Caused by: java.lang.ClassNotFoundException: cz.company.loom.host.epas.context.TransactionContextImpl from [Module "deployment.EpascustomerHostConnector-ear-1.11.9.1-host.ear.EpascustomerHostConnector-war-1.11.9.1.war:main" from Service Module Loader]

                                at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)

                                at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363)

                                at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351)

                                at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93)

                                ... 31 more

                             

                            All following occurences:

                             

                            2017-06-20 11:31:08,574 ERROR [io.undertow.request] (default task-181) UT005023: Exception handling request to /EpasHostConnector/host/xproto: java.lang.NoClassDefFoundError: cz/company/loom/host/epas/context/TransactionContextImpl

                                at cz.company.loom.host.epas.XProtoHostServlet.handleDocument(XProtoHostServlet.java:138)

                                at cz.company.loom.data.TransactionHandlerServletBase.doPost(TransactionHandlerServletBase.java:44)

                                at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

                                at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

                                at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

                                at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

                                at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

                                at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

                                at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

                                at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

                                at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

                                at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

                                at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

                                at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)

                                at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

                                at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

                                at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

                                at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

                                at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

                                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                                at java.lang.Thread.run(Thread.java:745)

                            • 11. Re: Random NoClassDefFoundError in WildFly
                              jaikiran pai Master

                              Caused by: java.lang.ClassNotFoundException: cz.company.loom.host.epas.context.TransactionContextImpl from [Module "deployment.EpascustomerHostConnector-ear-1.11.9.1-host.ear.EpascustomerHostConnector-war-1.11.9.1.war:main" from Service Module Loader]

                               

                              Looking at the war contents that you posted in one of your replies, the cz.company.loom.host.epas.context.TransactionContextImpl class isn't present in your .war file. Where exactly is that class located?

                              • 12. Re: Random NoClassDefFoundError in WildFly
                                Jiri Sebesta Newbie

                                It seems my previous comment was edited by somebody as it was very long. Here is the location of the class in the war (posting the whle package content):

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextException.class

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextBase.class

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContext.class

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextDiagnostic.class

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextImpl.class

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextDeclined.class

                                WEB-INF/classes/cz/company/loom/host/epas/context/TransactionContextFailed.class

                                 

                                I am pretty sure the class is in the package because the module works correctly after unassign+assing (and also before the first occurance).

                                • 13. Re: Random NoClassDefFoundError in WildFly
                                  jaikiran pai Master

                                  Jiri Sebesta wrote:

                                   

                                   

                                  I am pretty sure the class is in the package because the module works correctly after unassign+assing (and also before the first occurance).

                                  When you say unassign + assign, what do you mean? Are you talking about redeploying the application?

                                   

                                  Anyway, the best way to debug this is to enable TRACE level logging of org.jboss.modules package and see what's going on. It logs detailed messages about the module classloading that happens.

                                  • 14. Re: Random NoClassDefFoundError in WildFly
                                    Jiri Sebesta Newbie

                                    Yes, I mean redeploying the ear module. Thanks for the tip with trace logging, we wil try this.

                                    1 2 Previous Next