New ClassLoader initial checkin

[adrian.brock]

The new ClassLoader is in SVN.
https:/svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader
(for non jboss committers replace https://svn.jboss.org/... with http://anonsvn.jboss.org/
when the mirroring catches up).

First some housekeeping points.

1) Although I committed this to the microcontainer project, it doesn't really belong there.
It's only real dependency is the jboss logger.
I'm leaving it their for now while I work on the VFS/Deployer integration.
Eventually it should be moved to a top level project.

2) The original intention was to use the Felix ClassLoader and retrofit it with
UnifiedClassLoader/Repository like behaviour. In fact, it was easier to refactor
the UCL/R style classloader to add OSGI style behaviour.
The only things I've really used from the felix as the idea of delegate/filtered
classloaders and the hack you can find in ClassLoaderUtils::isRequestFromJDK()

This was mainly because the Felix classloader links directly into the OSGi
module system and more importantly doesn't include the fixes/workarounds
for the ClassCircularityError/deadlock problems.

[adrian.brock]

Design.

ClassLoaderPolicy

In the new ClassLoader, you don't really deal with classloaders at all.
You deal with ClassLoaderPolicys.
https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/ClassLoaderPolicy.java

This includes things like what packages you export, what you import, how to get
resources and other things related to defining classes.

ClassLoaderSystem and Domains

Additionally, the idea of a LoaderRepository is replaced with a ClassLoaderDomain
https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/ClassLoaderDomain.java
and a ClassLoaderSystem acting as a factory and a repository of domains.
https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/ClassLoaderSystem.java
Each ClassLoaderSystem has a "default domain".

Example psuedo code:

// NOTE: Singleton using the spi, but implementations can be instantiated
// directly to produce multiple classloading systems
// (which has proved useful in the tests)

ClassLoaderSystem system = ClassLoaderSystem.getInstance();

// Whatever
MyClassLoadPolicy policy = ...

// Register with the default domain, there are other methods to create
// and register with different domains
ClassLoader cl = system.registerClassLoaderPolicy(policy);

Uglyness and security

To remove some ugliness from the code, the spi is split into 2 parts.
The main part is what people will use
https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/
but these extend base implementations
https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/base/

The idea is that the base implementations use package private methods
to talk to each other about state, registration, etc. and issue well defined "callbacks"
into the main spi. The advantages being:

1) Removal of ugliness - you don't see "nitty gritty" in the main spi

2) Security - you can't break the spi nor can anybody get access to things
they shouldn't.

For the same reason, I've defined a lot of callbacks methods in the main spi
as protected since they should only be used as callbacks not by any random code.

[adrian.brock]

Some details

Loaders

A lot of the code is based on Loaders, well really DelegateLoaders.
https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/Loader.java
which do what you might expect.

Although I can support raw Loaders in some places (e.g. as parents of a Domain),
most places need DelegateLoaders. That is a Loader associated with a ClassLoaderPolicy
and transitively a ClassLoader.

DelegateLoaders

The DelegateLoaders are Loaders associated with a Policy.
https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/DelegateLoader.java
They can also implement additional rules. e.g. for handling imports or exports
you can add a filter.
https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/filter/FilteredDelegateLoader.java

ClassFilter

ClassFilters let you decide what parts of a delegate classloader you are going to
import or export.
https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/filter/ClassFilter.java
There are some standard ones defined like EVERYTHING and NOTHING in that class.

Examples.

A DelegateLoader that imports only a certain package

new FilteredDelegateLoader(policy, new PackageClassFilter(new String[] { "com.acme" });

Only importing java.* and javax.* from the parent of your domain

ClassLoadingSystem system = ClassLoadingSystem.getInstance();
system.createAndRegisterDomain("MyDomain", ParentPolicy.BEFORE_BUT_ONLY_JAVA);

See https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/ParentPolicy.java
for more on the parent policy.

Domains allow multiple hierarchies and not just delegation to another domain.

Loader parent = ...;
system.createAndRegisterDomain("MyDomain", ParentPolicy.BEFORE, parent);

this works because ClassLoadingDomains are also Loaders.

If you don't specify a parent then it uses the ClassLoader that defines
the org.jboss.classloader classes as the parent, via an adapter
https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/plugins/loader/ClassLoaderToLoaderAdapter.java

[adrian.brock]

"alesj" wrote:
I see that a couple of classes from .spi reference some of the .plugins classes.
I saw the same problem with my BeanMetaDataBuilder - moved impl details in the plugins.
Shouldn't we keep them separate -> only plugins depend on spi?

No. API/SPI is always allowed to reference plugins.
Otherwise factories wouldn't be able to define a default implementation for example.

The important part is that the implementation doesn't leak out to the client code, e.g.
by appearing as parameters on methods.

[adrian.brock]

NOTE: This thread is informational.

Please create a new thread for individual topics.
I don't want another 20-30 post discussion on different/unrelated topics.

[adrian.brock]

Back on topic.

So what is missing?

1) Policy

This ClassLoader is very dumb. It relies on you to tell it what the policy is.
The only current implementation of a policy is in the tests
https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/tests/org/jboss/test/classloader/support/MockClassLoaderPolicy.java

There are actually two more levels that need to be implemented.

a) VFS layer - write a policy that uses the VFS

b) Dependency and policy construction controller - i.e. from metadata workout
when dependencies are satisifed and create the relevant policies.
This will actually be part of the [Main]Deployer(s) when I plugin the MC to it.

2) Security

The current tests don't run with a security manager enabled, but also I need
to audit the classloader for two things:
a) Where privileged blocks are needed
b) What permissions we need to define, e.g. who can register policies or create domains

3) Caching/BlackListing

There is currenty no caching or black listing (cache misses).
This is going to be a little more complicated to implement since you can't
use a global cache for OSGi. It is no longer the "big ball mud" model.

The type and whether to cache should be part of the Policy.

4) Translater

I added a hook for translating classes, but I'm probably going to remove it
since this is already handled from Java5+ by the agents api in the JDK.

5) TODOs

There are a number of other TODOs and REVIEWs in the code, but these
are fairly minor.

6) Tests

I have some basic tests and I've ported some of the old classloader
tests from the jboss testsuite (where they didn't try to hook into the UCL).

Obviously more are needed. e.g. I have no tests at all for the domain
hierarchy.

[adrian.brock]

ClassLoaderManager

This is a replacement/port of LoadMgr3.

I originally tried to rewrite this as something intelligble but couldn't get it to work.

So I just copied it and simplified it, e.g. only the Domain deals with the
ClassLoadingTasks now and the ClassLoaderManger doesn't deal with domains
it deals with [Class]Loaders.

After I did this it still didn't work :-)

It was only when I implemented timeouts on the two most important waits

 boolean interrupted = thread.interrupted();

 int waits = 0;

 try
 {
 while (true)
 {
 try
 {
 if (lock.tryLock(0, TimeUnit.MICROSECONDS) == false)
 {
 // REVIEW: If we've been spinning for more than a minute then there is probably something wrong?
 if (waits++ == 6)
 throw new IllegalStateException("Waiting too long to get the classloader lock: " + this);
 // Wait 10 seconds
 if (trace)
 log.trace(this + " waiting for lock " + thread);
 this.wait(10000);
 }
 else
 {
 if (trace)
 log.trace(this + " aquiredLock " + thread + " holding=" + lock.getHoldCount());
 break;
 }
 }
 catch (InterruptedException ignored)
 {
 }
 }
 }
 finally
 {
 if (interrupted)
 thread.interrupt();
 }

that I figured out what was going on.
It started working with the timeouts, but only after a 10 second pause (co-incidence? :-)
A missed notification perhaps?

The answer was there is something subtle in the way the ClassLoaderManager
and the ClassLoader hand off control to each other.

Normally you release locks in the reverse order you aquire them.
Not here. Once I modified the lock release and notifyAll() to match the old
LoadMgr3 code it started working. :-)

Very Voodoo!

[adrian.brock]

I've updated the holder JIRA issue with subtasks
for things mentioned above:
http://jira.jboss.com/jira/browse/JBOSGI-7

[gcompienne]

Hello Adrian,

Early in the post you indicated that the classloading mechanism was in the microcontainer project but that it wasn't its rightfull place.

I wonder if your opinion has changed on this topic? I must admit from my point of view I have the tendency to see the microcontainer as the main mechanism allowing you to host components/containers together. And to me this means that classloading will typically play a critical role in this.

I agree that MC does not have to know the exact details (each subsystem could have its own way) but I still imagine that for things to work and cooperate the system would still need some general "rules of engagement". And these rules (things like the specification of the base CL policy mechanism) would have to be specified by MC.

Then when someone writes a container it would be clear that, as long as they respect the MC rules, then they have good chances that it will work.

Yes, things like classloaders could be separated from MC but then there will need to clarify the various projects that ones needs to be aware of in order to write a workable container.

I must admit this is probably a simplistic view (admin tasks comes to mind) and I am still trying to get my head round MC, but still I wonder if your opinion on this has changed or not.

Thanks.

Gilles.

[adrian.brock]

No I haven't changed my view.

The MC doesn't favour a specific classloader implementation.
It is left to the configuration how you create classloaders and what type they are.

There are some basic rules that need to be followed to work with what the other deployers
expect. e.g.
* support hierarchical domains/repositories
* switching from j2se classloading compliance
* delegation to peer classloaders
* allowing specific imports and versioning for OSGi style deployments
* (in future) filtering out certain packages, etc.

All these are encapsulated by the ClassLoaderMetaData.

If the classloader doesn't support these notions, (e.g. the old UCL doesn't
support OSGi version rules) then those deployments won't work as expected.

We could add validation to the legacy ServiceClassLoaderDeployer
that creates the old UCLs to spot when it cannot support the new
features, but in practice we'll probably just drop support for this old classloader
implementation.

So, in summary, I don't see a need to tightly integrate the classloader
with the MC. The question is only whether your choice of classloader
implementation can support all the features requested by the other
deployers via the ClassLoaderMetadata.

[gcompienne]

Ok thanks, this answers my question.

I didn't realize that your comment was only about the implementation itself (of the classloaders) and didn't include classes like ClassLoaderMetadata.

I agree that if the basic rules (hierarchical model,...) are followed then there should not be any issues.

Thanks for the clarification.

Gilles.