Need help understanding class loading rules used by JBoss Marshalling in Infinispan

[sellhorn]

This is related to using Infinispan in OSGi.

Basically, after some usage we sometimes see the JBoss Marshalling using the wrong class loaders in an OSGi environment. We have multiple bundles talking to a central bundle that holds all caches (the central bundle exposes itself as an OSGi service an does Dynamic-Import: *). Bundles have their own context and bundle class loaders and everything works fine. But when we run them all at the same time, after some time we start getting class loading exceptions and you can see jboss marshalling using the bundle context classloader from say bundle B, when it's trying to load say class A from bundle A.

It seems like the jboss marshaller keeps references to different classloaders and in a multithreaded environment can get messed up.

Here's an example stack trace of what I'm talking about;

Caused by: java.lang.ClassNotFoundException: com.ge.blah.A.SomeAClass not foundfrom bundle [com.ge.blah.B]

       at org.springframework.osgi.util.BundleDelegatingClassLoader.findClass(BundleDelegatingClassLoader.java:103)

       at org.springframework.osgi.util.BundleDelegatingClassLoader.loadClass(BundleDelegatingClassLoader.java:156)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:248)[:1.6.0_21]

       at java.lang.Class.forName0(Native Method)[:1.6.0_21]

       at java.lang.Class.forName(Class.java:247)[:1.6.0_21]

       at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:135)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:116)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.jboss.marshalling.river.RiverUnmarshaller.doResolveClass(RiverUnmarshaller.java:1148)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:862)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1183)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:270)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.jboss.marshalling.AbstractUnmarshaller.readObject(AbstractUnmarshaller.java:85)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:75)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.readObject(ConstantObjectTable.java:266)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.infinispan.marshall.jboss.ConstantObjectTable.readObject(ConstantObjectTable.java:253)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:357)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.jboss.marshalling.AbstractUnmarshaller.readObject(AbstractUnmarshaller.java:85)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:75)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.readObject(ConstantObjectTable.java:266)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.infinispan.marshall.jboss.ConstantObjectTable.readObject(ConstantObjectTable.java:253)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:357)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.jboss.marshalling.AbstractUnmarshaller.readObject(AbstractUnmarshaller.java:85)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.infinispan.marshall.jboss.GenericJBossMarshaller.objectFromObjectStream(GenericJBossMarshaller.java:162)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:111)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromByteBuffer(MarshallerAdapter.java:26)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:146)[198:com.ge.blah.someinfinispanwrapper:1.5.0.SNAPSHOT]

       ... 28 more

Caused by: java.lang.ClassNotFoundException: com.ge.blah.A.SomeAClass

       at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:506)[osgi-3.6.0.v20100517.jar:]

       at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:422)[osgi-3.6.0.v20100517.jar:]

       at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:410)[osgi-3.6.0.v20100517.jar:]

       at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)[osgi-3.6.0.v20100517.jar:]

       at java.lang.ClassLoader.loadClass(ClassLoader.java:248)[:1.6.0_21]

       at org.eclipse.osgi.internal.loader.BundleLoader.loadClass(BundleLoader.java:338)[osgi-3.6.0.v20100517.jar:]

       at org.eclipse.osgi.framework.internal.core.BundleHost.loadClass(BundleHost.java:232)[osgi-3.6.0.v20100517.jar:]

       at org.eclipse.osgi.framework.internal.core.AbstractBundle.loadClass(AbstractBundle.java:1197)[osgi-3.6.0.v20100517.jar:]

       at org.springframework.osgi.util.BundleDelegatingClassLoader.findClass(BundleDelegatingClassLoader.java:99)

       ... 56 more

One work around we've found is that if you turn on lazy deserialization then this seems to fix the problem, but we're seeing degraded performance and issues with running out of memory.

My question is, how does JBoss Marshalling figure out what class loader to use, and how I can make it use the classloader of the bundle that interacts with the class it loads ... OR ... how can I tell it what classloader to use in a more direct way. For example, I believe that if it used the classloader I have with Dynamic-Import: * things would work.

Are these some settings I can try for JBoss marshalling to control what class loader it uses, interceptors I could use?

Thanks

[sellhorn]

Forgot to mention, this is using Infinispan 4.1.0-FINAL

[galder.zamarreno]

Actually, forget what I said - this appears to be a limitation of the current classloading architechture. We're simply using the classloader associated with the current thread and that appears to have no access to another bundle.

[galder.zamarreno]

We've actually got a discussion going on about this on the dev forum that started with http://lists.jboss.org/pipermail/infinispan-dev/2011-April/007968.html and Pete summarised it in http://lists.jboss.org/pipermail/infinispan-dev/2011-May/008037.html

[dan.berindei]

I think what happens is that the Infinispan-created threads inherit the thread context classloader from the thread that first created getCache() on that CacheManager instance, which can be either the A classloader or the B classloader.

If I understand it correctly, Dynamic-Import will make the classes of both bundles to the central bundle's classloader, so you should be able to fix your problems by ensuring that the first getCache() call for any CacheManager instance happens while the TCCL is set to the central bundle's classloader.

Alternatively, Equinox has an extension that sets the TCCL to a special classloader that can access all the classes in the application: http://wiki.eclipse.org/Context_Class_Loader_Enhancements

In theory if you use this "buddy loading" mechanism you should have the same TCCL all the time, so the Infinispan threads should have the right TCCL regardless of how you create/access the cache.

[sellhorn]

Galder, Dan;

Thanks for the replies, very helpful information.

Basically what I've done so far is that in the wrapper class I have for infinispan datagrids, I have a pair of set/reset calls that will save the current TCCL, set it to the classloader of the "wrapper" class, touch the infinispan API and then reset the TCCL to whatever it was set before.

BTW, you need to do this on more than just getCache(), I had to add this to at least put and get. If I only do it on getCache() I get the same exception. To me, it seems like almost a concurrency issue with the marshaller.

One other thing I noticed, for bundles putting data of type A and B classes in one container, no problem. I was a bit surprised, I expected it to get messed up right at the start (this is a cluster of two machines). Now as soon as I start bundle A or B on the second node, it uses the wrong class loader ... I think it's on the put but didn't fully verify it.

Setting the class loader to the bundle with my wrapper class is not too horrible, because this class has a Dynamic-Import: *, which makes sense, it's a central OSGi friendly manager of datagrids in that environment. However, what should really be used is the class loader of the classes being put in the grid (A.class.getClassLoader()) but that would require tweaking code in the infinispan internals. Seems like from Galder's links, at least it is something that is being discussed.

BTW, what's the status of Infinispan in OSGi? I thought somebody was working specifically on making sure infinispan is OSGi friendly. Any word on this?

[galder.zamarreno]

Augusto, we have Luca working on getting Infinispan to be consumable as an OSGI bundle but some dependencies are not fully ready yet: https://issues.jboss.org/browse/ISPN-800