1 2 Previous Next 16 Replies Latest reply on Sep 19, 2017 12:42 PM by Michael Feldman

    JBoss and WildFly internal Lambda ClassNotFoundExceptions

    Michael Feldman Newbie

      I am working on upgrading our application from JBoss AS 6 to WildFly 10.1.0.CR1.

      When I exercise a particular action in our application, I get a ClassNotFoundException each time. But the missing class varies each time. They are all Jboss or WildFly classes that should be included with the distribution.

      Here are the ones I have seen so far:

       

      Caused by: java.lang.ClassNotFoundException: org.jboss.as.weld.services.bootstrap.WeldExecutorServices$$Lambda$113.1811321239 from [Module "org.jboss.as.weld:main" from local module loader @4ae82894 (finder: local module finder @543788f3 (roots: /apps/harmony/wildfly-10.1.0.CR1/modules,/apps/harmony/wildfly-10.1.0.CR1/modules/system/layers/base))] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)

       

      Caused by: java.lang.ClassNotFoundException: org.wildfly.clustering.web.infinispan.session.fine.SessionAttributeKeyExternalizer$$Lambda$64.44451544

       

      Caused by: java.lang.ClassNotFoundException: org.jboss.as.server.mgmt.ManagementWorkerService$$Lambda$46.495125226

       

      It is a Lambda class that is not found each time. Is that significant? The module jar files are always there when I check. So why can't they be found?

       

      Update:

      After my reply below, I searched all of the jar files included in my sar file for the classes that are not found. They do not include these classes. So how would WildFly be seeing the same class in two different jars?

        • 1. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
          gir489 Newbie

          I had the same problem. My particular problem was caused by having a hard dependency on the Undertow JARs, although before I had a dependency on the Tomcat JARs. It seems you're either double deploying the JBoss JARs, or are including some of your JBoss 6 JARs in your lib folder. That's my best guess. The $$somethingsomething means duplicate object instantiation from two different JARs, so it tries to create a unique identifier.

          • 2. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
            Michael Feldman Newbie

            Here are my jboss related Maven dependencies. I'm using a provided scope for all of them so I shouldn't be adding of my own jboss jars, right? Do you or does anyone else know if the same classes could be in some other libraries that wouldn't be as obvious? Is it a version problem? If I take these out I can't build my project.

             

                    <dependency>
                        <groupId>org.jboss.jbossts</groupId>
                        <artifactId>jbossjta</artifactId>
                        <version>4.16.6.Final</version>
                        <type>jar</type>
                        <scope>provided</scope>
                    </dependency>
                    <dependency>
                        <groupId>org.jboss.spec.javax.ejb</groupId>
                        <artifactId>jboss-ejb-api_3.1_spec</artifactId>
                        <version>1.0.2.Final</version>
                        <type>jar</type>
                        <scope>provided</scope>
                    </dependency>
                    <dependency>
                        <groupId>org.jboss.spec</groupId>
                        <artifactId>jboss-javaee-all-6.0</artifactId>
                        <version>3.0.3.Final</version>
                        <type>jar</type>
                        <scope>provided</scope>
                    </dependency>
                    <dependency>
                        <groupId>org.jboss.ejb3</groupId>
                        <artifactId>jboss-ejb3-ext-api</artifactId>
                        <version>2.1.0</version>
                        <scope>provided</scope>
                    </dependency>
                    <dependency>
                        <groupId>org.jboss</groupId>
                        <artifactId>jboss-vfs</artifactId>
                        <version>3.1.0.Final</version>
                        <scope>provided</scope>
                    </dependency>
            
            • 3. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
              gir489 Newbie

                   Maven is just a tool that pulls the JARs when built. You'd need to check either your Wildfly deployments folder or your app's lib folder.

              • 4. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
                Radoslav Husar Master

                Right, the provided scope is correct. I would suggest inspecting the resulting ear/war for unintentionally bundled dependency jars.

                • 5. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
                  Michael Feldman Newbie

                  Please see my Update in my main post. I searched all of the jar files included in my sar file for the classes that are not found. They do not include these classes. I also searched all of the jars in the WildFly modules directory. Each of the classes not found above are only listed in one jar file. I don't know where else WildFly could be finding another version of these classes.

                  • 6. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
                    gir489 Newbie

                    I don't know what else to tell you. The error it's complaining about is that during runtime that are multiple copies of the same .class file being wired in from multiple locations, so the JVM is getting confused. You're going to have to debug the application and figure out why.

                    • 7. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
                      David Lloyd Master

                      The $$somethingsomething means duplicate object instantiation from two different JARs, so it tries to create a unique identifier.

                      This is not necessarily the case.  Lambda invocation classes are generated using identifiers like this; seeing these in the stack trace is generally not (by itself) an indicator of a packaging problem.

                      • 8. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
                        David Lloyd Master

                        Michael Feldman wrote:

                         

                        I am working on upgrading our application from JBoss AS 6 to WildFly 10.1.0.CR1.

                        When I exercise a particular action in our application, I get a ClassNotFoundException each time. But the missing class varies each time. They are all Jboss or WildFly classes that should be included with the distribution.

                        Here are the ones I have seen so far:

                         

                        Caused by: java.lang.ClassNotFoundException: org.jboss.as.weld.services.bootstrap.WeldExecutorServices$$Lambda$113.1811321239 from [Module "org.jboss.as.weld:main" from local module loader @4ae82894 (finder: local module finder @543788f3 (roots: /apps/harmony/wildfly-10.1.0.CR1/modules,/apps/harmony/wildfly-10.1.0.CR1/modules/system/layers/base))] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)

                         

                        Caused by: java.lang.ClassNotFoundException: org.wildfly.clustering.web.infinispan.session.fine.SessionAttributeKeyExternalizer$$Lambda$64.44451544

                         

                        Caused by: java.lang.ClassNotFoundException: org.jboss.as.server.mgmt.ManagementWorkerService$$Lambda$46.495125226

                         

                        It is a Lambda class that is not found each time. Is that significant? The module jar files are always there when I check. So why can't they be found?

                         

                        Update:

                        After my reply below, I searched all of the jar files included in my sar file for the classes that are not found. They do not include these classes. So how would WildFly be seeing the same class in two different jars?

                        Can you share the JDK version string of the JDK you are running on?

                        • 9. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
                          Michael Feldman Newbie

                          OpenJDK Runtime Environment (build 1.8.0_141-b16)

                          • 10. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
                            Michael Feldman Newbie

                            And I just downloaded Wildfly-10.1.0.Final to see if I would see different results. But no, same thing as Wildfly-10.1.0.CR1.

                            • 11. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
                              jaikiran pai Master

                              To make it easier to narrow down the issue, I would like to suggest a few things:

                               

                              1. Try this against latest 11.0.0.CR1 release. If that's not feasible, please use/stick to 10.1.0.Final

                              2. Post the complete exception stacktrace that you see

                              3. Explain a bit about your application packaging and what technologies it uses. Also, what triggers this exception - is it triggered just by deploying the application or is it triggered when you invoke something?

                              4. How are you deploying the application - is any IDE involved in _deploying_ the application or starting the server?

                              • 12. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
                                David Lloyd Master

                                Michael Feldman wrote:

                                 

                                OpenJDK Runtime Environment (build 1.8.0_141-b16)

                                Maybe this sounds a bit like a copout, but could you give it a try using Oracle's JDK?  These lambda-related CNFE/NCDFE kinds of problems look very fishy to me.

                                • 13. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
                                  Michael Feldman Newbie

                                  I spent the morning trying some things and gathering more information. I didn't realize until you asked that I wasn't using the Oracle JDK. I'm using a different user to run the application to more closely match our deployment environments but I didn't realize it wasn't pointing to the right JDK. I switched to the Oracle JDK:

                                   

                                  Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

                                   

                                  but that didn't fix the problem. I switched to WildFly-11.0.0.CR1 but that did not fix the problem. But now I am getting a different Lambda class that isn't found. And it is consistently the same one each time now. I will post the full stack trace below. The rits Cloner is in the stack trace. I was wondering if that was related but it wasn't there each time before. But now it is.

                                   

                                  I am deploying a SAR file using the JBoss CLI tool.

                                   

                                  Before the operation we are performing, we make a deepClone of the object so we can compare them later after the object is updated with new data. You can see in the stack trace below where this is happening. The object being cloned in this case is a list of objects. The objects just contain String, longs, Dates, an enum, etc. Nothing too out of the ordinary.

                                   

                                  Here is the stack trace:

                                   

                                  Caused by: java.lang.NoClassDefFoundError: java/util/function/Consumer$$Lambda$346/457772971
                                  at sun.reflect.GeneratedSerializationConstructorAccessor586.newInstance(Unknown Source)
                                  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                                  at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:56)
                                  at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:73)
                                  at com.rits.cloning.ObjenesisInstantiationStrategy.newInstance(ObjenesisInstantiationStrategy.java:18)
                                  at com.rits.cloning.Cloner.newInstance(Cloner.java:287)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:464)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:458)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:458)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:458)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.FastClonerHashMap.clone(FastClonerHashMap.java:21)
                                  at com.rits.cloning.Cloner.fastClone(Cloner.java:108)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:436)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
                                  at com.rits.cloning.Cloner.deepClone(Cloner.java:323)
                                  at com.metronaviation.harmony.ansp.common.util.BeanDeltaMonitor.before(BeanDeltaMonitor.java:236)
                                  at com.metronaviation.ips.processor.data.util.AirportDeltaMonitor.before(AirportDeltaMonitor.java:115)
                                  at com.metronaviation.ips.processor.data.DeltaDataUpdateMonitorImpl.startMonitorBeforeProcess(DeltaDataUpdateMonitorImpl.java:193)
                                  at com.metronaviation.ips.processor.data.DataModificationBean.handleModification(DataModificationBean.java:323)
                                  at sun.reflect.GeneratedMethodAccessor398.invoke(Unknown Source)
                                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                  at java.lang.reflect.Method.invoke(Method.java:498)
                                  at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
                                  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
                                  at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
                                  at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90)
                                  at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
                                  at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
                                  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
                                  at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
                                  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
                                  at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
                                  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
                                  at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
                                  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
                                  at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
                                  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
                                  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
                                  at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
                                  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
                                  at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
                                  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
                                  at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
                                  ... 71 more
                                  • 14. Re: JBoss and WildFly internal Lambda ClassNotFoundExceptions
                                    David Lloyd Master

                                    Ah I see.  This may be an Objenesis bug (see https://github.com/kostaskougios/cloning/issues/45) where the cloner tries to clone the method handle's internal lambda instance which essentially cannot be cloned for a variety of reasons.  Maybe switch to a serialization-based strategy instead?

                                    1 2 Previous Next