Scoping of war/jar file embedded in sar service archives

[adinn]

I just identified an 'interesting' change in processing of sar files between AS 4.X/5.0.BetaX and As 5.0.CR1. Here's the setup:

I have a sar which embeds 8 different wars. The listeners configured in the respective web.xml files are implemented in 8 different jars also embedded in the sar. All of the listeners refer to a class Sequencer which is supplied in one of the 8 embedded jars. This class is used to sequence callbacks registered by the lsiteners (I know, I could probably do this using a jboss web.xml but that's another story and besides ...)

In the old AS versions the listeners all saw the same copy of class Sequencer. Ok, specifically, modifications to static data defined by this class were visible to all listeners so I guess that means they are using the same class. In 5.0.CR1 they all get their own copy of the class -- well at least I assume that is what is happening. What I have observed is that they are all seeing their own version of the static fields of Sequencer.

So that kind of snookers my sequencing code. Rather a drastic change to the semantics I feel. Any comments on whether this was intended or accidental? Can we go back to the previous behaviour?

.

[adinn]

Sorry, just to clarify -- it is the listener implementations which refer to the common class, Sequencer.

[jhalliday]

Hello JBossAS people.

I'd appreciate it if we could get some feedback on this one sooner rather than later - it's holding up the JBossTS release. Is there a config workaround that gives us the beta4 behaviour back?

Thanks.

[adinn]

Hi guys,

I have found a workaround for this problem. If unjar all my jar code and place the resulting classes tree in the sar at the top level then the static data defined in class Sequencer is shared by all the listener implementations.

[starksm64]

I'll look at this today. I see Ales added some class loading tests but don't know if they cover this case yet. What is your deployment issue Jonathan?

[adinn]

Hi Scott,

Thanks for looking at this. The problem was explained rather clumsily in the original post. Here is a picture of the setup which might make things clearer:

example.sar
 war1.war
 WEB-INFO/web.xml
 ...
 <listener>com.arjuna.Listener1</listener>
 ...
 war2.war
 WEB-INFO/web.xml
 ...
 <listener>com.arjuna.Listener2</listener>
 ...
 jar1.jar
 com/arjuna/Listener1.class
 com/arjuna/Sequencer.class
 ...
 jar2.jar
 com/arjuna/Listener2.class
 ...

Listener1 and Listener2 both refer to static array in class Sequencer. However, they appear to see two different versions of this array. Listener1 is supposed to see changes made by Listener2 and vice versa but this does not happen. I assume this is because they are each referring to different versions of the class and that this is something to do with the way the war classloader interacts with the sar/jar loaders. If I unjar the classes and place them in the sar at top level then the listeners all see the same version of Sequencer i.e. this works ok:

example.sar
 war1.war
 WEB-INFO/web.xml
 ...
 <listener>com.arjuna.Listener1</listener>
 ...
 war2.war
 WEB-INFO/web.xml
 ...
 <listener>com.arjuna.Listener2</listener>
 ...
 com/arjuna/Listener1.class
 com/arjuna/Sequencer.class
 com/arjuna/Listener2.class
 ...

[starksm64]

I have not been able to reproduce this with a sar like the following:

[569][valkyrie: testsuite]$ jar -tf output/lib/staticarray2.sar
META-INF/
META-INF/MANIFEST.MF
org/
org/jboss/
org/jboss/test/
org/jboss/test/util/
org/jboss/test/util/Debug.class
staticarray-web1.war
staticarray-web2.war
staticarray1.jar
staticarray2.jar
[570][valkyrie: testsuite]$ jar -tf output/lib/staticarray-web1.warWEB-INF/
WEB-INF/web.xml
validate.jsp
[571][valkyrie: testsuite]$ jar -tf output/lib/staticarray-web2.war
WEB-INF/
WEB-INF/web.xml
validate.jsp
[572][valkyrie: testsuite]$ jar -tf output/lib/staticarray1.jar
META-INF/
META-INF/MANIFEST.MF
org/
org/jboss/
org/jboss/test/
org/jboss/test/classloader/
org/jboss/test/classloader/sharing/
org/jboss/test/classloader/sharing/staticarray/
org/jboss/test/classloader/sharing/staticarray/common/
org/jboss/test/classloader/sharing/staticarray/common/AbstractServletRequestListener.class
org/jboss/test/classloader/sharing/staticarray/common/Listener1.class
org/jboss/test/classloader/sharing/staticarray/common/Sequencer.class
org/jboss/test/classloader/sharing/staticarray/common/SequencerMBean.class
[573][valkyrie: testsuite]$ jar -tf output/lib/staticarray2.jar
META-INF/
META-INF/MANIFEST.MF
org/
org/jboss/
org/jboss/test/
org/jboss/test/classloader/
org/jboss/test/classloader/sharing/
org/jboss/test/classloader/sharing/staticarray/
org/jboss/test/classloader/sharing/staticarray/common/
org/jboss/test/classloader/sharing/staticarray/common/AbstractServletRequestListener.class
org/jboss/test/classloader/sharing/staticarray/common/Listener2.class

Do you have a jboss-web.xml in the wars affecting the class loading setup?

[adinn]

Ok, Scott, I found the cause of the problem -- there were a couple of small but significant details in the sar setup which I failed to include in the problem report.

Firstly, the jars are actually layered as 8 war-specific jars which employ code from 5 generic code jars. Sequencer, the class with the static data, was in one of the generic jars (ws-c.jar).

Secondly, the war files and war-specific jar files employ Classpath entries in the manifest to refer to the jars they rely on. So, for example, ws-c11.war refers to ws-c11.jar which refers to ws-c.jar (contains Sequencer), while ws-t11.war refers to ws-t11.jar and ws-c11.jar and ws-t11.jar refers to ws-t.jar and ws-c.jar.

I worked out by trial and error that removing the classpath entries in the war file manifest solves the class duplication problem. I assume this is because the war class loader grabs its own copy of classes in the the jars (recursively) mentioned in the classpath rather than delegating to the sar loader.

I did not actually need to remove the jar classpath entries to fix the problem and assume this would not be strictly necessary since the entries only reference jars contained within the same sar i.e. they do not intercede between the loader of the referencing class and some other loader. Is that a valid assumption or was I just lucky? I'll still remove the classpath entries anyway -- the app works fine without them.

I don't know whether this needs to be documented as a bug, change of behaviour or just useful thing someone might want to know some day. Anyway, it probably ought to be noted somewhere where AS users or the support team can find it (well, it is noted here at least :-)

[starksm64]

"adinn" wrote:

I worked out by trial and error that removing the classpath entries in the war file manifest solves the class duplication problem. I assume this is because the war class loader grabs its own copy of classes in the the jars (recursively) mentioned in the classpath rather than delegating to the sar loader.

Yes, the default war model is to use any class in its classpath before delegating to the parent unless either the web deployer or a jboss-web.xml changes this.

"adinn" wrote:

I did not actually need to remove the jar classpath entries to fix the problem and assume this would not be strictly necessary since the entries only reference jars contained within the same sar i.e. they do not intercede between the loader of the referencing class and some other loader. Is that a valid assumption or was I just lucky? I'll still remove the classpath entries anyway -- the app works fine without them.

Its doubtful the jar classpath entries have any affect as the referenced jars are already part of the deployment classpath. No reason to remove them.

"adinn" wrote:

I don't know whether this needs to be documented as a bug, change of behaviour or just useful thing someone might want to know some day. Anyway, it probably ought to be noted somewhere where AS users or the support team can find it (well, it is noted here at least :-)

I just want to understand what the difference is. It sounds like the war manifest classpath behavior has changed, but if anything I would expect that in jbossas5 the war manifest was ignored while what you describe appears to be the opposite; that 4.X/5.0.BetaX were not using the war manifest classpath. I think we have gone around on whether the war manifest classpath should be picked up or not.

At this point I just want to reproduce the issue in a unit test. If I can't tweak my current example I'll need to get your deployment to see what exactly is happening.

[adinn]

At this point I just want to reproduce the issue in a unit test. If I can't tweak my current example I'll need to get your deployment to see what exactly is happening.

Ok, well I think you should be able to reproduce the behaviour I saw if you modify the example you cited earlier as follows:

add a manifest file to staticarray-web1.war which refers to staticarray1.jar via a classpath entry
add a manifest file to staticarray-web2.war which refers to staticarray2.jar via a classpath entry
add a manifest file to staticarray2.jar which mentions staticarray1.jar via a classpath entry

it may be possible to do it with a shorter route

add a manifest file to staticarray-web1.war which refers to staticarray1.jar via a classpath entry
add a manifest file to staticarray-web2.war which refers to staticarray2.jar and staticarray1.jar via a classpath entry

if this fails to err . . . fail . . . and you need my code let me know

[adrian.brock]

I just want to understand what the difference is. It sounds like the war manifest classpath behavior has changed, but if anything I would expect that in jbossas5 the war manifest was ignored while what you describe appears to be the opposite; that 4.X/5.0.BetaX were not using the war manifest classpath. I think we have gone around on whether the war manifest classpath should be picked up or not.

The issue is probably that the war manifest classpath should only be processed
when it is a top level deployment (well nearly see below :-).

If it is not a top level deployment, then the jars will already be included
in the top level deployment classloader (and should be shared
between the wars).

This logic would go in the WARStructure deployer

 // Add the war manifest classpath entries (if its top level)
 if (isTopLevel(parent))
 addClassPath(root, file, false, true, context);

However it should really also check (for non-top level deployments)
whether the manifest entry is included as a context within the parent.

e.g. an ear might not delcare the jar in application.xml,
instead it is only referenced from the war manifest.

 // Add the war manifest classpath entries (if its top level)
 if (isTopLevel(parent) || doesntAlreadyHaveContext(metadata, file))
 addClassPath(root, file, false, true, context);

If you agree with my analysis, then create a JIRA issue for it.

[starksm64]

Yes, the issue is JBAS-5773 and the test which shows the behavior using the staticarray.sar with:

staticarray1.jar
staticarray2.jar
staticarray-web1.war referring to staticarray1.jar via a classpath entry
staticarray-web2.war referring staticarray2.jar and staticarray1.jar via a classpath entry

I'll create a deployers issue as well for the WARStructure change.

[starksm64]

jira is down so I'll create it when its back up.

[adrian.brock]

http://jira.jboss.com/jira/browse/JBDEPLOY-59

Actually, I think the StructureDeployer is probably not the correct place
to add this behavour?

The issue is that we now allow greater control of what/how subdeployment classloaders
get constructed. Even to the extent that a subdeployment classloader
may not be able to see the parent deployment classloader.

e.g.
top-level: domain=foo parent=DefaultDomain
sub-deployment: domain=bar parent=DefaultDomain

Although, I'm not sure somebody would really want to do the above :-)

It would mean that sub-deployment's classloaders needs the manifest entry
even though it is already included in the top-level.

So I think the correct place to put this code would be in the usage of
ClassPathVisitor to "weed out" duplicates in sub-deployment classloader
that are already in one of the parent classloaders (if the parent deployment's classloader
is really a parent of the sub-deployment).

[adrian.brock]

The reason for not doing it in the structure determination is that
we don't know the classloading rules until much later.

Parsing jboss-classloading.xml in the structure deployers is not an option
because the rules can come from legacy deployment descriptors
and anybody could write their own deployer to create or modify the
classloading metadata in an ad hoc anyway.