Overriding server libs in an EJB inside an EAR - 5.1.0.GA

[rodrigo.uchoa]

Hi everyone!

This is something I've been struggling for quite some time. That is, how to change the classloader behavior in an EAR deploy. I've read a lot of posts and docs around the internet, but nothing seems to be exactly my case.

It all started with the need to update Hibernate libs in JBoss 5.1.0.GA (from the bundled 3.2.4 to 3.3.1). Using the jboss-classloading.xml file it worked fine when we use a standalone .WAR deploy with the hibernate libs inside "WEB-INF/lib" directory. When we switched to an .EAR deploy, it stopped working. Our .EAR structure looks like this:

EAR

|

|---- lib //all libs, including hibernate's, are here

|

|---- META-INF

|             |---- jboss-classloading.xml

|

|---- EJB_Module //persistence-unit defined here, along with EJB business classes

|

|---- WAR_Module

              |---- WEB-INF

                          |---- jboss-classloading.xml

The EAR jboss-classloading.xml:

<classloading xmlns="urn:jboss:classloading:1.0"
    domain="app.ear"
    export-all="NON_EMPTY"
    import-all="true"
    parent-first="false">
</classloading>

The WAR's jboss-classloading.xml:

<classloading xmlns="urn:jboss:classloading:1.0"
    domain="app.war"
    parent-domain="app.ear"
    export-all="NON_EMPTY"
    import-all="true">
</classloading>

Now to the problem... Let's suppose I have two different .EAR projects with the exact same structure described above. When I deploy both of them to the same JBoss AS instance, one of them always have classloading issues with the hibernate libs when server is starting up. More specifically, we get "ClassCastException" with a hibernate interceptor we have:

Caused by: javax.persistence.PersistenceException: [PersistenceUnit: sinpi-pu] Interceptor class does not implement Interceptor interface: br.gov.dpf.sinpi.core.util.HistoricoInterceptor
    at org.hibernate.ejb.Ejb3Configuration.configure(Ejb3Configuration.java:811)
    at org.hibernate.ejb.Ejb3Configuration.configure(Ejb3Configuration.java:191)
    at org.hibernate.ejb.Ejb3Configuration.configure(Ejb3Configuration.java:253)
    at org.hibernate.ejb.HibernatePersistence.createEntityManagerFactory(HibernatePersistence.java:125)
    at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:52)
    at org.jboss.seam.persistence.EntityManagerFactory.createEntityManagerFactory(EntityManagerFactory.java:85)
    at org.jboss.seam.persistence.EntityManagerFactory.startup(EntityManagerFactory.java:50)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:592)
    at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)
    at org.jboss.seam.util.Reflections.invokeAndWrap(Reflections.java:144)
    at org.jboss.seam.Component.callComponentMethod(Component.java:2283)
    at org.jboss.seam.Component.callCreateMethod(Component.java:2198)
    at org.jboss.seam.Component.newInstance(Component.java:2158)
    ... 75 more
Caused by: java.lang.ClassCastException: br.gov.dpf.sinpi.core.util.HistoricoInterceptor
    at org.hibernate.ejb.Ejb3Configuration.configure(Ejb3Configuration.java:793)

My assumption is that different "org.hibernate.Interceptor" classes loaded by different classloaders are causing this mess. This problem only occurs when we deploy two EAR's at the same time. I even tried to change the "ear-deployer-jboss-beans.xml" isolated configuration, but it didn't work. What I need is that both EAR's use the newer hibernate libs bundled inside each "ear/lib" directory.

Any ideas would be appreciated.

[alesj]

You're saying it works with a single .ear?

(as I think it should with that your .ear/META-INF/jboss-classloading.xml)

When deploying 2 .ears,

do you use the same domain name in both jboss-classloading.xml?

btw: I think you don't need that jboss-classloading.xml in .war

[rodrigo.uchoa]

It works with a single .EAR. And yes, the domain names are different in each .EAR

[alesj]

It works with a single .EAR.

Hmm, really strange why it wouldn't then work with two.

Debug? To see where it pulls that HibernateInterceptor class.

And yes, the domain names are different in each .EAR

Just making sure. :-)

[rodrigo.uchoa]

How exactly should I debug this? What should I look for? I'm kinda newbie with these classloading issues.

[alesj]

In your code, you could do something like:

System.out.println("HI::classloader >> " + HibernateInterceptor.class.getClassLoader());

Or remote debug of JBossAS and stopping the code in Ejb3Configuration.configure(Ejb3Configuration.java:793)

then doing the same, somehow. :-)

[rodrigo.uchoa]

Sorry for the delay. I had to leave work before I could answer you yesterday.

So, the debugging turned out to show something really strange. This piece of code I'm debugging, inside Ejb3Configuration.class:

               Class interceptor = classForName( interceptorName );

                cfg.setInterceptor( (Interceptor) interceptor.newInstance() );

According to the stack trace, I get a ClassCastException in line two. As if my Hibernate interceptor does not implement the "org.hibernate.Interceptor" interface, which it does. Like I said before, I have two EAR's being deployed, and in only one of them I have this custom Interceptor. Here's the output from the debug:

//Ejb3Configuration classloader

this.getClass().getClassLoader()

     (org.jboss.classloader.spi.base.BaseClassLoader) BaseClassLoader@ac97e2{vfsfile:/Users/rodrigouchoa/Java/jboss-5.1.0-dpf.GA/server/default/deploy/corporativo-packaging.ear/}

//My custom interceptor classloader (its different)   

interceptor.getClassLoader()

     (org.jboss.classloader.spi.base.BaseClassLoader) BaseClassLoader@40bd2e{vfsfile:/Users/rodrigouchoa/Java/jboss-5.1.0-dpf.GA/server/default/deploy/sinpi-packaging.ear/}

//The hibernate classloader (same as the first one)      

org.hibernate.Interceptor.class.getClassLoader()

     (org.jboss.classloader.spi.base.BaseClassLoader) BaseClassLoader@ac97e2{vfsfile:/Users/rodrigouchoa/Java/jboss-5.1.0-dpf.GA/server/default/deploy/corporativo-packaging.ear/}

It seems that my custom interceptor is coming from the right classloader, inside the EAR it belongs to. But the Ejb3Configuration.class is running inside the other EAR's classloader! The first and last output are from "corporativo-packaging.ear", and the middle one is from "sinpi-packaging.ear" . How can I isolate these two EARs?

The java2ParentDelegation seems to be working okay. The problem is isolating one EAR from the other.

[alesj]

How can I isolate these two EARs?

Different domain name should already separate them.

[rodrigo.uchoa]

Different domain name should already separate them.

But it didn't.

Here's the two jboss-classloading files (each inside ear/META-INF)

<?xml version="1.0" encoding="UTF-8"?>

<classloading xmlns="urn:jboss:classloading:1.0"

domain="corporativo.ear"

export-all="NON_EMPTY"

import-all="true"

parent-first="false">

</classloading>

<?xml version="1.0" encoding="UTF-8"?>

<classloading xmlns="urn:jboss:classloading:1.0"

domain="sinpi.ear"

export-all="NON_EMPTY"

import-all="true"

parent-first="false">

</classloading>

The first domain is "corporativo.ear" and the second is "sinpi.ear". Any more clues? Thanks anyway.

[rodrigo.uchoa]

Ales,

Doesn't it have something to do with the domain the EJB modules inside each EAR? My custom interceptor is inside its EAR ejb module. Shouldnt it have its own jboss-classloading.xml file?

[alesj]

Doesn't it have something to do with the domain the EJB modules inside each EAR? My custom interceptor is inside its EAR ejb module. Shouldnt it have its own jboss-classloading.xml file?

I doubt it.

EJBs are probably in your sub-deployment (x.ear/my-ejbs.jar), which belongs to root CL (as one of its roots).

So same rules should apply.

[rodrigo.uchoa]

Ales Justin wrote:

I doubt it.

EJBs are probably in your sub-deployment (x.ear/my-ejbs.jar), which belongs to root CL (as one of its roots).

So same rules should apply.

I see.

Does anyone have any ideas?

This happens with all EAR's... We can only deploy one EAR per JBoss 5.1.0.GA AS instance. Everytime two or more of them are deployed at the same time, this type of classloading issues with hibernate libs arise (all EARs have their own hibernate libs inside them, in ear/lib). Unfortunately, we need to update hibernate libs. Is there any other way to achieve the same goal?

[alesj]

Can you debug a call from BaseClassLoader::loadClass("<the problematic class name>")?

And perhaps see why it gets picked-up from the wrong classloader.

[rodrigo.uchoa]

Didn't find anything unusual. I turned on the "TRACE" level for org.jboss.classloader.spi.base.BaseClassLoader. It shows hibernate libs being loaded by the two different domains, as it should. The question remains... Why is one EAR using the other EAR's libs when starting up?

If anyone ever come up with any ideas... Please share.

[alesj]

I mean, you should attach a remote debuger to your AS instance.

This is a pita, as you need to have all the code -- AS project has it, and we use Maven, so it's doable.

Then step-by-step, to see why the classloader from one domain actually jumps / sees the other domain, where they should be isolated from each-other.

Make sure you don't have any unneccessary jboss-classloading.xml files present, which could instruct the CL to point to wrong domain.