JBoss and WildFly internal Lambda ClassNotFoundExceptions

[mikedevva]

I am working on upgrading our application from JBoss AS 6 to WildFly 10.1.0.CR1.

When I exercise a particular action in our application, I get a ClassNotFoundException each time. But the missing class varies each time. They are all Jboss or WildFly classes that should be included with the distribution.

Here are the ones I have seen so far:

Caused by: java.lang.ClassNotFoundException: org.jboss.as.weld.services.bootstrap.WeldExecutorServices$$Lambda$113.1811321239 from [Module "org.jboss.as.weld:main" from local module loader @4ae82894 (finder: local module finder @543788f3 (roots: /apps/harmony/wildfly-10.1.0.CR1/modules,/apps/harmony/wildfly-10.1.0.CR1/modules/system/layers/base))] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)

Caused by: java.lang.ClassNotFoundException: org.wildfly.clustering.web.infinispan.session.fine.SessionAttributeKeyExternalizer$$Lambda$64.44451544

Caused by: java.lang.ClassNotFoundException: org.jboss.as.server.mgmt.ManagementWorkerService$$Lambda$46.495125226

It is a Lambda class that is not found each time. Is that significant? The module jar files are always there when I check. So why can't they be found?

Update:

After my reply below, I searched all of the jar files included in my sar file for the classes that are not found. They do not include these classes. So how would WildFly be seeing the same class in two different jars?

[gir489]

I had the same problem. My particular problem was caused by having a hard dependency on the Undertow JARs, although before I had a dependency on the Tomcat JARs. It seems you're either double deploying the JBoss JARs, or are including some of your JBoss 6 JARs in your lib folder. That's my best guess. The $$somethingsomething means duplicate object instantiation from two different JARs, so it tries to create a unique identifier.

[mikedevva]

Here are my jboss related Maven dependencies. I'm using a provided scope for all of them so I shouldn't be adding of my own jboss jars, right? Do you or does anyone else know if the same classes could be in some other libraries that wouldn't be as obvious? Is it a version problem? If I take these out I can't build my project.

        <dependency>
            <groupId>org.jboss.jbossts</groupId>
            <artifactId>jbossjta</artifactId>
            <version>4.16.6.Final</version>
            <type>jar</type>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.jboss.spec.javax.ejb</groupId>
            <artifactId>jboss-ejb-api_3.1_spec</artifactId>
            <version>1.0.2.Final</version>
            <type>jar</type>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.jboss.spec</groupId>
            <artifactId>jboss-javaee-all-6.0</artifactId>
            <version>3.0.3.Final</version>
            <type>jar</type>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.jboss.ejb3</groupId>
            <artifactId>jboss-ejb3-ext-api</artifactId>
            <version>2.1.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.jboss</groupId>
            <artifactId>jboss-vfs</artifactId>
            <version>3.1.0.Final</version>
            <scope>provided</scope>
        </dependency>

[gir489]

     Maven is just a tool that pulls the JARs when built. You'd need to check either your Wildfly deployments folder or your app's lib folder.

[rhusar]

Right, the provided scope is correct. I would suggest inspecting the resulting ear/war for unintentionally bundled dependency jars.

[mikedevva]

Please see my Update in my main post. I searched all of the jar files included in my sar file for the classes that are not found. They do not include these classes. I also searched all of the jars in the WildFly modules directory. Each of the classes not found above are only listed in one jar file. I don't know where else WildFly could be finding another version of these classes.

[gir489]

I don't know what else to tell you. The error it's complaining about is that during runtime that are multiple copies of the same .class file being wired in from multiple locations, so the JVM is getting confused. You're going to have to debug the application and figure out why.

[dmlloyd]

The $$somethingsomething means duplicate object instantiation from two different JARs, so it tries to create a unique identifier.

This is not necessarily the case.  Lambda invocation classes are generated using identifiers like this; seeing these in the stack trace is generally not (by itself) an indicator of a packaging problem.

[dmlloyd]

Michael Feldman wrote:

I am working on upgrading our application from JBoss AS 6 to WildFly 10.1.0.CR1.

When I exercise a particular action in our application, I get a ClassNotFoundException each time. But the missing class varies each time. They are all Jboss or WildFly classes that should be included with the distribution.

Here are the ones I have seen so far:

Caused by: java.lang.ClassNotFoundException: org.jboss.as.weld.services.bootstrap.WeldExecutorServices$$Lambda$113.1811321239 from [Module "org.jboss.as.weld:main" from local module loader @4ae82894 (finder: local module finder @543788f3 (roots: /apps/harmony/wildfly-10.1.0.CR1/modules,/apps/harmony/wildfly-10.1.0.CR1/modules/system/layers/base))] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)

Caused by: java.lang.ClassNotFoundException: org.wildfly.clustering.web.infinispan.session.fine.SessionAttributeKeyExternalizer$$Lambda$64.44451544

Caused by: java.lang.ClassNotFoundException: org.jboss.as.server.mgmt.ManagementWorkerService$$Lambda$46.495125226

It is a Lambda class that is not found each time. Is that significant? The module jar files are always there when I check. So why can't they be found?

Update:

After my reply below, I searched all of the jar files included in my sar file for the classes that are not found. They do not include these classes. So how would WildFly be seeing the same class in two different jars?

Can you share the JDK version string of the JDK you are running on?

[mikedevva]

OpenJDK Runtime Environment (build 1.8.0_141-b16)

[mikedevva]

And I just downloaded Wildfly-10.1.0.Final to see if I would see different results. But no, same thing as Wildfly-10.1.0.CR1.

[jaikiran]

To make it easier to narrow down the issue, I would like to suggest a few things:

1. Try this against latest 11.0.0.CR1 release. If that's not feasible, please use/stick to 10.1.0.Final

2. Post the complete exception stacktrace that you see

3. Explain a bit about your application packaging and what technologies it uses. Also, what triggers this exception - is it triggered just by deploying the application or is it triggered when you invoke something?

4. How are you deploying the application - is any IDE involved in _deploying_ the application or starting the server?

[dmlloyd]

Michael Feldman wrote:

OpenJDK Runtime Environment (build 1.8.0_141-b16)

Maybe this sounds a bit like a copout, but could you give it a try using Oracle's JDK?  These lambda-related CNFE/NCDFE kinds of problems look very fishy to me.

[mikedevva]

I spent the morning trying some things and gathering more information. I didn't realize until you asked that I wasn't using the Oracle JDK. I'm using a different user to run the application to more closely match our deployment environments but I didn't realize it wasn't pointing to the right JDK. I switched to the Oracle JDK:

Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

but that didn't fix the problem. I switched to WildFly-11.0.0.CR1 but that did not fix the problem. But now I am getting a different Lambda class that isn't found. And it is consistently the same one each time now. I will post the full stack trace below. The rits Cloner is in the stack trace. I was wondering if that was related but it wasn't there each time before. But now it is.

I am deploying a SAR file using the JBoss CLI tool.

Before the operation we are performing, we make a deepClone of the object so we can compare them later after the object is updated with new data. You can see in the stack trace below where this is happening. The object being cloned in this case is a list of objects. The objects just contain String, longs, Dates, an enum, etc. Nothing too out of the ordinary.

Here is the stack trace:

Caused by: java.lang.NoClassDefFoundError: java/util/function/Consumer$$Lambda$346/457772971
at sun.reflect.GeneratedSerializationConstructorAccessor586.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:56)
at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:73)
at com.rits.cloning.ObjenesisInstantiationStrategy.newInstance(ObjenesisInstantiationStrategy.java:18)
at com.rits.cloning.Cloner.newInstance(Cloner.java:287)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:464)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:458)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:458)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:458)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.FastClonerHashMap.clone(FastClonerHashMap.java:21)
at com.rits.cloning.Cloner.fastClone(Cloner.java:108)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:436)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.cloneInternal(Cloner.java:487)
at com.rits.cloning.Cloner.deepClone(Cloner.java:323)
at com.metronaviation.harmony.ansp.common.util.BeanDeltaMonitor.before(BeanDeltaMonitor.java:236)
at com.metronaviation.ips.processor.data.util.AirportDeltaMonitor.before(AirportDeltaMonitor.java:115)
at com.metronaviation.ips.processor.data.DeltaDataUpdateMonitorImpl.startMonitorBeforeProcess(DeltaDataUpdateMonitorImpl.java:193)
at com.metronaviation.ips.processor.data.DataModificationBean.handleModification(DataModificationBean.java:323)
at sun.reflect.GeneratedMethodAccessor398.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:90)
at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)
at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
... 71 more

[dmlloyd]

Ah I see.  This may be an Objenesis bug (see https://github.com/kostaskougios/cloning/issues/45) where the cloner tries to clone the method handle's internal lambda instance which essentially cannot be cloned for a variety of reasons.  Maybe switch to a serialization-based strategy instead?