JBoss 4.0.1 Linkage Error Nightmare

[rhook]

We have a very real need to support the following situation:

- a server will have a JBoss 4.x installation, possibly with certain 3rd party libraries such as iBatis in the {server}/lib directory;
- we will deploy multiple different EAR files for our various internal applications;
- some of those EAR files will contain different versions of the same classes, as the applications will be at different versions;
- we must have each EAR file isolated from all others in terms of loading the classes that are within the EAR file, but "common" 3rd party libraries will be sourced from the {server}/lib directory;

I have spent many (many (many (many))) hours fighting with class loader problems trying to support this situation, with many (many (many (many))) hours spent reading through these forums, the support wiki, google-sphere at large, and the (free) jboss documentation.

As far as I can determine, the blessed and correct way to provide the required "isolation with overloading" I describe is as specified in http://www.jboss.org/wiki/Wiki.jsp?page=ClassLoadingConfiguration

Question 1: is that the correct method?
Question 1a: if not, what is the correct method?

With regard to the following I have been testing by shutting down Jboss, removing the deploy ear, completely hosing out {server}/tmp/deploy each time, starting up Jboss, then deploying the EAR.

I have found the following situations, for different jboss-app.xml in the EAR file. Note that the exact class that is complained about varies in our case between javax.sql.Datasource and javax.xml.transformSource, depending which of two particular test EAR I used.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE jboss-app PUBLIC "-//JBoss//DTD J2EE Application 1.4//EN" "http://www.jboss.org/j2ee/dtd/jboss-app_4_0.dtd">
<jboss-app>
 <loader-repository>
 au.com.hpa.AuthServer:loader=AuthServerApp.ear
 <loader-repository-config>
 java2ParentDelegation=false
 </loader-repository-config>
 </loader-repository>
</jboss-app>

results in "java.lang.LinkageError: loader constraints violated when linking javax/xml/transformSource class"

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE jboss-app PUBLIC "-//JBoss//DTD J2EE Application 1.4//EN" "http://www.jboss.org/j2ee/dtd/jboss-app_4_0.dtd">
<jboss-app>
 <loader-repository loaderRepositoryClass="org.jboss.mx.loading.HeirarchicalLoaderRepository3">
 au.com.hpa.AuthServer:loader=AuthServerApp.ear
 <loader-repository-config configParserClass="org.jboss.mx.loading.HeirarchicalLoaderRepository3ConfigParser">
 java2ParentDelegation=false
 </loader-repository-config>
 </loader-repository>
</jboss-app>

results in "java.lang.LinkageError: loader constraints violated when linking javax/xml/transformSource class"

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE jboss-app PUBLIC "-//JBoss//DTD J2EE Application 1.4//EN" "http://www.jboss.org/j2ee/dtd/jboss-app_4_0.dtd">
<jboss-app>
 <loader-repository>
 au.com.hpa.AuthServer:loader=AuthServerApp.ear
 <loader-repository-config>
 java2ParentDelegation=true
 </loader-repository-config>
 </loader-repository>
</jboss-app>

results in "java.lang.NoClassDefFoundError: org/apache/digester/ObjectCreationFactory"

Question 2: for our desired approach, should java2ParentDelegation be true or false;

and there is probably a meta-question here:

Question N: where, if anywhere, is the correct and unambiguous documentation?

I will now turn on "bitter complaint" mode: a lot of people seem to have had very similar problems in the past, but almost universally if they have found a resolution, it has not been posted. The documentation is sorely lacking here as well: most of it appears to be hugely out of date, and the 4.x documentation is incomplete. And before anyone points me at the PDF document that describes the linkage error, yes, it contains an indepth discussion of the class loader model, an in-depth explanation of what a linkage error is, a fair amount of test code to show what is going on when a linkage error occurs, and contains no concrete explanation of how to overcome the problem. The closest thing to a work-around that I've seen posted in any number of instances has been to include all requisite JAR in the EAR or the WAR inside the ear, however this is not suitable in this case because:

- it prevents sharing of common 3rd party libraries;
- the "requisite" JAR in this case would be things like rt.jar from the JDK!

This issue has become a major show-stopper for us, and is a source of immense frustration. We have explicitly made the decision to move from another application server to JBoss because it claims to be J2EE compliant, and this was precisely the problem we were unable to resolve in the other application server without running multiple instances of the server.

All sensible (concrete, tested, verified) suggestions and responses eagerly and urgently sought.

[rhook]

The wiki page http://www.jboss.org/wiki/Wiki.jsp?page=JBoss4FAQ is incorrect where it states

In the default configuration, the ear-deployer.xml is configured to isolate deployments

In fact the ear-deployer.xml in the default configuration as deployed is:

<?xml version="1.0" encoding="UTF-8"?>

<!-- The JBoss service configuration file for the EAR deployer service.
$Id: ear-deployer.xml,v 1.1 2004/09/05 15:50:06 starksm Exp $
-->
<server>
 <!-- EAR deployer, remove if you are not using ear deployments -->
 <mbean code="org.jboss.deployment.EARDeployer"
 name="jboss.j2ee:service=EARDeployer">
 <attribute name="Isolated">false</attribute>
 <attribute name="CallByValue">false</attribute>
 </mbean>
</server>

In my previous posting, removing the isolation settings from jboss-app.xml and flipping the two boolean attributes in ear-deployer.xml resulted in exactly the same difficulty: linkage errors for classes in rt.jar

[rhook]

In my previous, as far as I can determine all classes for which linkage errors are being reported are coming out of rt.jar, possibly limited to those in the javax.* package.

[rhook]

This may or may not be an appropriate solution. Re-examing the 4.0.1 documentation, I found in chapter 9, section 9.1, discussion of configuring the tomcat 5.x bundled inside Jboss 4.0.1. It discusses the contents of the jboss-service.xml file for Tomcat 5, and says

UseJBossWebLoader: This flag indicates that Tomcat should use a JBoss unified class loader as the web application class loader. The default is true, which means that the classes inside of the WEB-INF/classes and WEB-INF/lib directories of the WAR file are incorporated into the default shared class loader repository described in Chapter 2, The JBoss JMX Microkernel. This may not be what you want as its contrary to the default servlet class loading model and can result in sharing of classes/resources between web applications. You can disable this by setting this attribute to false.

Making this change appears to have resulted in the effect that we desire: classes within separate EAR are loaded separately. It appears that by making this change, the alterations to jboss-app.xml for each EAR is not required, but of course it makes a global change for the server (which is what we need, but your mileage may vary).

I'm still waiting for, and hoping that, someone from jboss.org will respond to my first post in this thread, to clarify the issues that I've raised. I strongly suspect that either the documentation is wildly incorrect, or JBoss 4.0.1 does not actually support isolation with overloading as described in the Wiki. Is it possible this is an error that was introduced with 4.0.1? If I have time, I will download 4.0.0 and see whether the behaviour is any different there.

[jgra]

All JBoss classloading scenarios are available on following wiki page:
http://www.jboss.org/wiki/Wiki.jsp?page=JBossClassLoadingUseCases
Try to test them on your server before deploy of your real applications.
BTW, You should never ever add rt.jar to ear archives, it's completely wrong.

Regards
Janusz

[starksm64]

Class loading isolation with overrides is the correct approach, but the deployment should have have any java.* or javax.* classes in the deployment as these cannot be overriden. Apparently most other class loading approaches just ignore attempts to load such classes and so don't care about their existence. Currently we don't have a notion of classes that cannot be loaded from the deployment so they need to be removed to work with scoped class loading.

[starksm64]

UseJBossWebLoader only affects the wars in the ear, not the ear itself. When set to false you are using the tomcat class loader as a child to the ear class loader instead of using the ear unified loader at the war layer.

[rhook]

"jgra" wrote:
All JBoss classloading scenarios are available on following wiki page:
http://www.jboss.org/wiki/Wiki.jsp?page=JBossClassLoadingUseCases
Try to test them on your server before deploy of your real applications.
BTW, You should never ever add rt.jar to ear archives, it's completely wrong.

Regards
Janusz

Yes, thank you. I had read that material prior to posting, and was indeed aware that putting rt.jar in the EAR is completely wrong. You may notice that the material you reference in the WIKI does not contain a concrete example of the requisite settings in jboss-app.xml. I refer to my original post: my testing (locally, on a test server, before deploying real applications) suggests that the recommended approach is resulting in linkage errors for classes in rt.jar.

[rhook]

UseJBossWebLoader only affects the wars in the ear, not the ear itself. When set to false you are using the tomcat class loader as a child to the ear class loader instead of using the ear unified loader at the war layer.

Thanks for that clarification, that was the understanding that I had gained, but it is nice to see it explicitly stated that way. For our purposes, short term, that is a suitable situation. Longer term, I would prefer to move away from it.

Class loading isolation with overrides is the correct approach, but the deployment should have have any java.* or javax.* classes in the deployment as these cannot be overriden. Apparently most other class loading approaches just ignore attempts to load such classes and so don't care about their existence. Currently we don't have a notion of classes that cannot be loaded from the deployment so they need to be removed to work with scoped class loading.

Ok, call me a Bear of Very Little Brain {http://www.just-pooh.com}, but that has confused me. Correct me if I'm wrong here, you are confirming that i should be altering jboss-app.xml to specify a loader repository, and turning java2parentDelegation off, however you are also saying that the classes for which i am getting linkage errors will need to be explicitly available in the EAR, even though they appear to be classes that should be available in rt.jar?

As an aside, I fully intend to add some documentation regarding the resolution of this problem to the wiki, just as soon as i can actually sign in to the wiki, and the issue is resolved.

[rhook]

Let me rephrase the question, in the hope that somebody, anybody will respond:

a) should java2parentdelegation in jboss-app.xml be true or false;
b) how do we ensure that the java.* and javax.* classes in rt.jar are accesible?

I'll reiterate: I'm not simply trying to leach free resources here, I'm genuinely intending to post somewhere the resolution for this particular use case, as soon as a resolution can be devised. I'll also emphasise: I'm not knocking JBoss, whinging about the UCL, or complaining about democrats/republicans/venusians. However, I'm either completely stupid and cannot see an obvious solution, or I've horribly misinterpreted the documentation, or the documentation simply does not contain the solution. (Or an error was introduced between 4.0.0 and 4.0.1, that's a scenario I've not had the time to explore yet)

[starksm64]

The jboss-app.xml shown with the java2ParentDelegation=false as stated in the docs is the correct configuration for scoped class loading with isolation.

The rt.jar is always accessible due to every class loader inheriting the system class loader. You cannot override any classes in rt.jar from the jboss level.

[rhook]

Thank you for that clarification Scott, I shall continue wrestling with this. I had indicated in a previous post that I wanted to explore whether this was new behaviour in 4.0.1 compared to 4.0.0. To that end, I did the following. Note that the server configuration used in 4.0.1 was a copy of the "default" configuration, with the jboss-ws4ee.sar package removed. This was done because we are using jbuilder 2005, and there is a different issue related to some profound weirdness around axis, meaning that (a) we have to ship the axis libraries jbuilder is using, (b) any other axis libraries are verboten. Apart from that, the 4.0.1 server definition we are using is a vanilla definition, straight out of the box. Oops, I tell a lie. The 4.0.1 server definition has had the UseJBossWebLoader set to FALSE in the tomcat jboss-service.xml.

1. shipped EAR to 4.0.1, worked ok. This had no specification of a classloader in it's jboss-app.xml;
2. shipped to 4.0.0, horrible axis failures on invocation of a method;
3. stopped 4.0.0, slapped forehead, removed jboss-ws4ee.sar, restarted, axis problem went away, product worked ok;
4. set UseJBossWebLoader to TRUE in 4.0.1, shipped same EAR, horrible failures on deploy;

I then explored the differences between the 4.0.0 config and 4.0.1 config, and found the following for the "default" server def:

- 4.0.1 has deployables not present for 4.0.0: scheduler-service.xml, schedule-manager-servers.xml, properties-service.xml, jboss-hibernate.deployer, jboss-aop.deployer, htti-invoker.sar, mail-ra.rar, cache-invalidation-service.xml

- 4.0.0 has deployables not present for 4.0.1: ebxmlrr-service.sar, hibernate-deployer-service.xml, iiop-service.xml, jboss-jdbc-metadata.sar
(note that the 4.0.0 "standard" deploy is different to the 4.0.1 "default" as well)

Config differences for tomcat jboss-service.xml:

4.0.0 "default" has UseJBossWebLoader set to FALSE
4.0.0 "standard" has UseJBossWebLoader set toTRUE
4.0.1 "default" has UseJBossWebLoader set to TRUE

Config diffrences for ear-deployer.xml:
4.0.0 "default" has Isolated and CallByValue set to TRUE
4.0.0 "standard" has Isolated and CallByValue set to FALSE
4.0.1 "default" has Isolated and CallByValue set to FALSE

Now, this makes it appear as though the 4.0.0 "standard" is equivalent to the 4.0.1 "default". This was no great suprise, as that's what the 4.0.1 release notes say. It does mean though that folk who are not experiencing these problems with 4.0.0 who are using the "default" server (which is probably almost all of them) are getting isolation at the WAR level because deployed WAR are just using the tomcat class loader.

The next, obvious thing for me to try was to make a copy of the 4.0.0 "standard" server, remove jboss-ws4ee.sar, and deploy the same EAR. Which I did, and to nobody's suprise it had the same horrible deploy problems as for our copy of the 4.0.1 "default" server.

This was kind of re-assuring, as it means that I don't have to start doing diffs on the code to see if some breakage occurred between 4.0.0 and 4.0.1 (I have my own mountains of code to maintain, and get going urgently, thank you, without taking on any other mountains of code for the time being)

Where to from here? Well, based on Scott's response, this means that I will setup a configuration with the tomcat UseJBossWebLoader set to "true", and put the following in my EAR:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE jboss-app PUBLIC "-//JBoss//DTD J2EE Application 1.4//EN" "http://www.jboss.org/j2ee/dtd/jboss-app_4_0.dtd">
<jboss-app>
 <loader-repository>
 au.com.hpa.banking:loader=RemittanceProcessingApp.ear
 <loader-repository-config>
 java2ParentDelegation=false
 </loader-repository-config>
 </loader-repository>
</jboss-app>

and leave Isolated and CallByValue in ear-deployer.xml set to FALSE. Stay tuned for more exciting instalments.