10 Replies Latest reply on Jan 31, 2008 8:38 AM by Gilles Compienne

    New ClassLoader initial checkin

    Adrian Brock Master

      The new ClassLoader is in SVN.
      https:/svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader
      (for non jboss committers replace https://svn.jboss.org/... with http://anonsvn.jboss.org/
      when the mirroring catches up).

      First some housekeeping points.

      1) Although I committed this to the microcontainer project, it doesn't really belong there.
      It's only real dependency is the jboss logger.
      I'm leaving it their for now while I work on the VFS/Deployer integration.
      Eventually it should be moved to a top level project.

      2) The original intention was to use the Felix ClassLoader and retrofit it with
      UnifiedClassLoader/Repository like behaviour. In fact, it was easier to refactor
      the UCL/R style classloader to add OSGI style behaviour.
      The only things I've really used from the felix as the idea of delegate/filtered
      classloaders and the hack you can find in ClassLoaderUtils::isRequestFromJDK()

      This was mainly because the Felix classloader links directly into the OSGi
      module system and more importantly doesn't include the fixes/workarounds
      for the ClassCircularityError/deadlock problems.

        • 1. Re: New ClassLoader initial checkin
          Adrian Brock Master

          Design.

          ClassLoaderPolicy

          In the new ClassLoader, you don't really deal with classloaders at all.
          You deal with ClassLoaderPolicys.
          https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/ClassLoaderPolicy.java

          This includes things like what packages you export, what you import, how to get
          resources and other things related to defining classes.

          ClassLoaderSystem and Domains

          Additionally, the idea of a LoaderRepository is replaced with a ClassLoaderDomain
          https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/ClassLoaderDomain.java
          and a ClassLoaderSystem acting as a factory and a repository of domains.
          https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/ClassLoaderSystem.java
          Each ClassLoaderSystem has a "default domain".

          Example psuedo code:

          // NOTE: Singleton using the spi, but implementations can be instantiated
          // directly to produce multiple classloading systems
          // (which has proved useful in the tests)
          
          ClassLoaderSystem system = ClassLoaderSystem.getInstance();
          
          // Whatever
          MyClassLoadPolicy policy = ...
          
          // Register with the default domain, there are other methods to create
          // and register with different domains
          ClassLoader cl = system.registerClassLoaderPolicy(policy);
          


          Uglyness and security

          To remove some ugliness from the code, the spi is split into 2 parts.
          The main part is what people will use
          https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/
          but these extend base implementations
          https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/base/

          The idea is that the base implementations use package private methods
          to talk to each other about state, registration, etc. and issue well defined "callbacks"
          into the main spi. The advantages being:

          1) Removal of ugliness - you don't see "nitty gritty" in the main spi

          2) Security - you can't break the spi nor can anybody get access to things
          they shouldn't.

          For the same reason, I've defined a lot of callbacks methods in the main spi
          as protected since they should only be used as callbacks not by any random code.

          • 2. Re: New ClassLoader initial checkin
            Adrian Brock Master

            Some details

            Loaders

            A lot of the code is based on Loaders, well really DelegateLoaders.
            https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/Loader.java
            which do what you might expect.

            Although I can support raw Loaders in some places (e.g. as parents of a Domain),
            most places need DelegateLoaders. That is a Loader associated with a ClassLoaderPolicy
            and transitively a ClassLoader.

            DelegateLoaders

            The DelegateLoaders are Loaders associated with a Policy.
            https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/DelegateLoader.java
            They can also implement additional rules. e.g. for handling imports or exports
            you can add a filter.
            https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/filter/FilteredDelegateLoader.java

            ClassFilter

            ClassFilters let you decide what parts of a delegate classloader you are going to
            import or export.
            https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/filter/ClassFilter.java
            There are some standard ones defined like EVERYTHING and NOTHING in that class.

            Examples.

            A DelegateLoader that imports only a certain package

            new FilteredDelegateLoader(policy, new PackageClassFilter(new String[] { "com.acme" });
            


            Only importing java.* and javax.* from the parent of your domain
            ClassLoadingSystem system = ClassLoadingSystem.getInstance();
            system.createAndRegisterDomain("MyDomain", ParentPolicy.BEFORE_BUT_ONLY_JAVA);
            

            See https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/ParentPolicy.java
            for more on the parent policy.

            Domains allow multiple hierarchies and not just delegation to another domain.
            Loader parent = ...;
            system.createAndRegisterDomain("MyDomain", ParentPolicy.BEFORE, parent);
            

            this works because ClassLoadingDomains are also Loaders.

            If you don't specify a parent then it uses the ClassLoader that defines
            the org.jboss.classloader classes as the parent, via an adapter
            https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/plugins/loader/ClassLoaderToLoaderAdapter.java

            • 3. Re: New ClassLoader initial checkin
              Adrian Brock Master

               

              "alesj" wrote:
              I see that a couple of classes from .spi reference some of the .plugins classes.
              I saw the same problem with my BeanMetaDataBuilder - moved impl details in the plugins.
              Shouldn't we keep them separate -> only plugins depend on spi?


              No. API/SPI is always allowed to reference plugins.
              Otherwise factories wouldn't be able to define a default implementation for example.

              The important part is that the implementation doesn't leak out to the client code, e.g.
              by appearing as parameters on methods.

              • 4. Re: New ClassLoader initial checkin
                Adrian Brock Master

                NOTE: This thread is informational.

                Please create a new thread for individual topics.
                I don't want another 20-30 post discussion on different/unrelated topics.

                • 5. Re: New ClassLoader initial checkin
                  Adrian Brock Master

                  Back on topic.

                  So what is missing?

                  1) Policy

                  This ClassLoader is very dumb. It relies on you to tell it what the policy is.
                  The only current implementation of a policy is in the tests
                  https://svn.jboss.org/repos/jbossas/projects/microcontainer/trunk/classloader/src/tests/org/jboss/test/classloader/support/MockClassLoaderPolicy.java

                  There are actually two more levels that need to be implemented.

                  a) VFS layer - write a policy that uses the VFS

                  b) Dependency and policy construction controller - i.e. from metadata workout
                  when dependencies are satisifed and create the relevant policies.
                  This will actually be part of the [Main]Deployer(s) when I plugin the MC to it.

                  2) Security

                  The current tests don't run with a security manager enabled, but also I need
                  to audit the classloader for two things:
                  a) Where privileged blocks are needed
                  b) What permissions we need to define, e.g. who can register policies or create domains

                  3) Caching/BlackListing

                  There is currenty no caching or black listing (cache misses).
                  This is going to be a little more complicated to implement since you can't
                  use a global cache for OSGi. It is no longer the "big ball mud" model.

                  The type and whether to cache should be part of the Policy.

                  4) Translater

                  I added a hook for translating classes, but I'm probably going to remove it
                  since this is already handled from Java5+ by the agents api in the JDK.

                  5) TODOs

                  There are a number of other TODOs and REVIEWs in the code, but these
                  are fairly minor.

                  6) Tests

                  I have some basic tests and I've ported some of the old classloader
                  tests from the jboss testsuite (where they didn't try to hook into the UCL).

                  Obviously more are needed. e.g. I have no tests at all for the domain
                  hierarchy.

                  • 6. Re: New ClassLoader initial checkin
                    Adrian Brock Master

                    ClassLoaderManager

                    This is a replacement/port of LoadMgr3.

                    I originally tried to rewrite this as something intelligble but couldn't get it to work.

                    So I just copied it and simplified it, e.g. only the Domain deals with the
                    ClassLoadingTasks now and the ClassLoaderManger doesn't deal with domains
                    it deals with [Class]Loaders.

                    After I did this it still didn't work :-)

                    It was only when I implemented timeouts on the two most important waits

                     boolean interrupted = thread.interrupted();
                    
                     int waits = 0;
                    
                     try
                     {
                     while (true)
                     {
                     try
                     {
                     if (lock.tryLock(0, TimeUnit.MICROSECONDS) == false)
                     {
                     // REVIEW: If we've been spinning for more than a minute then there is probably something wrong?
                     if (waits++ == 6)
                     throw new IllegalStateException("Waiting too long to get the classloader lock: " + this);
                     // Wait 10 seconds
                     if (trace)
                     log.trace(this + " waiting for lock " + thread);
                     this.wait(10000);
                     }
                     else
                     {
                     if (trace)
                     log.trace(this + " aquiredLock " + thread + " holding=" + lock.getHoldCount());
                     break;
                     }
                     }
                     catch (InterruptedException ignored)
                     {
                     }
                     }
                     }
                     finally
                     {
                     if (interrupted)
                     thread.interrupt();
                     }
                    


                    that I figured out what was going on.
                    It started working with the timeouts, but only after a 10 second pause (co-incidence? :-)
                    A missed notification perhaps?

                    The answer was there is something subtle in the way the ClassLoaderManager
                    and the ClassLoader hand off control to each other.

                    Normally you release locks in the reverse order you aquire them.
                    Not here. Once I modified the lock release and notifyAll() to match the old
                    LoadMgr3 code it started working. :-)

                    Very Voodoo!

                    • 7. Re: New ClassLoader initial checkin
                      Adrian Brock Master

                      I've updated the holder JIRA issue with subtasks
                      for things mentioned above:
                      http://jira.jboss.com/jira/browse/JBOSGI-7

                      • 8. Re: New ClassLoader initial checkin
                        Gilles Compienne Newbie

                        Hello Adrian,

                        Early in the post you indicated that the classloading mechanism was in the microcontainer project but that it wasn't its rightfull place.

                        I wonder if your opinion has changed on this topic? I must admit from my point of view I have the tendency to see the microcontainer as the main mechanism allowing you to host components/containers together. And to me this means that classloading will typically play a critical role in this.

                        I agree that MC does not have to know the exact details (each subsystem could have its own way) but I still imagine that for things to work and cooperate the system would still need some general "rules of engagement". And these rules (things like the specification of the base CL policy mechanism) would have to be specified by MC.

                        Then when someone writes a container it would be clear that, as long as they respect the MC rules, then they have good chances that it will work.

                        Yes, things like classloaders could be separated from MC but then there will need to clarify the various projects that ones needs to be aware of in order to write a workable container.

                        I must admit this is probably a simplistic view (admin tasks comes to mind) and I am still trying to get my head round MC, but still I wonder if your opinion on this has changed or not.

                        Thanks.

                        Gilles.

                        • 9. Re: New ClassLoader initial checkin
                          Adrian Brock Master

                          No I haven't changed my view.

                          The MC doesn't favour a specific classloader implementation.
                          It is left to the configuration how you create classloaders and what type they are.

                          There are some basic rules that need to be followed to work with what the other deployers
                          expect. e.g.
                          * support hierarchical domains/repositories
                          * switching from j2se classloading compliance
                          * delegation to peer classloaders
                          * allowing specific imports and versioning for OSGi style deployments
                          * (in future) filtering out certain packages, etc.

                          All these are encapsulated by the ClassLoaderMetaData.

                          If the classloader doesn't support these notions, (e.g. the old UCL doesn't
                          support OSGi version rules) then those deployments won't work as expected.

                          We could add validation to the legacy ServiceClassLoaderDeployer
                          that creates the old UCLs to spot when it cannot support the new
                          features, but in practice we'll probably just drop support for this old classloader
                          implementation.

                          So, in summary, I don't see a need to tightly integrate the classloader
                          with the MC. The question is only whether your choice of classloader
                          implementation can support all the features requested by the other
                          deployers via the ClassLoaderMetadata.

                          • 10. Re: New ClassLoader initial checkin
                            Gilles Compienne Newbie

                            Ok thanks, this answers my question.

                            I didn't realize that your comment was only about the implementation itself (of the classloaders) and didn't include classes like ClassLoaderMetadata.

                            I agree that if the basic rules (hierarchical model,...) are followed then there should not be any issues.

                            Thanks for the clarification.

                            Gilles.