10 Replies Latest reply on Aug 3, 2010 5:40 AM by kabirkhan

    Benchmarking classloaders

    kabirkhan

      For AS 6 we might want to look at setting up imports and exports for the classloaders, which might be faster than using the current ImportAll/Big-ball-of-mud setting. However, that sounds like a very big task, so I we should benchmark this in isolation to see if imports/exports will actually be faster. I have something simple working for ImportAll, but need to think a bit about how to do this with imports and exports.

        • 1. Re: Benchmarking classloaders
          bosschaert

          Hi Kabir,

           

          I just implemented something like this for the OSGi performance benchmarks. Here's a quick description of the bundle dependency topology that it creates:

          5 (Versioned) Common Bundles
             - Exports org.jboss.osgi.test.common;version=x
          5 Numbered (but not versioned) Util Bundles
             - Imports org.jboss.osgi.test.common
             - Exports org.jboss.osgi.test.util[x];uses="org.jboss.osgi.test.common"
          5 Versioned Interfaces Bundles
             - Exports org.jboss.osgi.test.versioned;version=x
          5 Versioned Impl Bundles
             - Imports org.jboss.osgi.test.common;version=[x,x]
             - Imports org.jboss.osgi.test.versioned;version=[x,x]
             - Imports org.jboss.osgi.test.util[x]
             - Exports org.jboss.osgi.test.versioned.impl;version=x;uses=org.jboss.osgi.test.util[x]
          a large number of test bundles (number configurable)
             - Imports org.jboss.osgi.test.common;version=[x,x]
             - Imports org.jboss.osgi.test.versioned;version=[x,x]
             - Imports org.jboss.osgi.test.versioned.impl;version=[x,x]

          Where x is a number [1..5].

           

          Each test bundle loads a class of each of its 3 dependency packages in its activator. This also triggers an indirect load on the Util[x] class.

           

          It's available on master in jbosgi on github: BundleInstallAndStartBenchmark.java

           

          You might be able to reuse some of this...

           

          David

          • 2. Re: Benchmarking classloaders
            kabirkhan
            I have got some basic benchmarks working. Instead of using AS classes, I am now generating my own using Chiba's ClassFileWriter to have better control over what exists where. The classes are only generated if the target/generated-classes and target/generated-jars directories don't exist already. You tell it how many jars to create, how many packages per jar and how many classes per package. If you change these numbers, be sure to delete the target/generated-* directories. So, if 2 is selected for each of these it then creates the following jars:
            interface0.jar:
            org.jboss.test.interface0.pkg0.Interface0.class
            org.jboss.test.interface0.pkg0.Interface1.class
            org.jboss.test.interface0.pkg1.Interface0.class
            org.jboss.test.interface0.pkg1.Interface1.class
             
             
            interface1.jar:
            org.jboss.test.interface1.pkg0.Interface0.class
            org.jboss.test.interface1.pkg0.Interface1.class
            org.jboss.test.interface1.pkg1.Interface0.class
            org.jboss.test.interface1.pkg1.Interface1.class
             
             
            abstract0.jar:
            org.jboss.test.abstract0.pkg0.Abstract0.class
            org.jboss.test.abstract0.pkg0.Abstract1.class
            org.jboss.test.abstract0.pkg1.Abstract0.class
            org.jboss.test.abstract0.pkg1.Abstract1.class
             
             
            abstract1.jar:
            org.jboss.test.abstract1.pkg0.Abstract0.class
            org.jboss.test.abstract1.pkg0.Abstract1.class
            org.jboss.test.abstract1.pkg1.Abstract0.class
            org.jboss.test.abstract1.pkg1.Abstract1.class
             
             
            impl0.jar:
            org.jboss.test.impl0.pkg0.Impl0.class
            org.jboss.test.impl0.pkg0.Impl1.class
            org.jboss.test.impl0.pkg1.Impl0.class
            org.jboss.test.impl0.pkg1.Impl1.class
             
             
            impl1.jar:
            org.jboss.test.impl1.pkg0.Impl0.class
            org.jboss.test.impl1.pkg0.Impl1.class
            org.jboss.test.impl1.pkg1.Impl0.class
            org.jboss.test.impl1.pkg1.Impl1.class
            
            An example of the expected inheritance hierarchies is:
            class org.jboss.test.impl1.pkg1.Impl1 extends org.jboss.test.abstract1.pkg1.Abstract1{}
            class org.jboss.test.abstract1.pkg1.Abstract1 implements org.jboss.test.interface1.pkg1.Interface1{}
            
            I then create a classloader for each jar, with different classloading rules per test and try to load the Impl classes from the implx.jar loaders, which triggers searches for the superclasses/interfaces in the other classloaders.
            Ales, can you take a look at how I am setting up the VFSClassLoaderFactories? I do this in the TestCase classes. The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.
            Running this a few times on a set of 50 jars with 10 packages per jar and 10 classes per package, I get the following results.:

             

            Import/Export packages:
            Deploying the VFSClassLoaderFactories and creating the loaders: 1862ms 1693ms 1847ms 1725ms 1706ms
            Loading the classes: 5008ms 4883ms 4744ms 4893ms 5012ms
             
             
            Export/ImportAll:
            Deploying the VFSClassLoaderFactories and creating the loaders: 1657ms 1627ms 1659ms 1720ms 1573ms
            Loading the classes: 4858ms 4693ms 4796ms 4714ms 4899ms
             
             
            Import/Export module:
            Deploying the VFSClassLoaderFactories and creating the loaders: 1785ms 1736ms 1766ms 1819ms 1901ms
            Loading the classes: 4992ms 5609ms 5510ms 4991ms 5626ms
            
            
            • 3. Re: Benchmarking classloaders
              alesj
              Ales, can you take a look at how I am setting up the VFSClassLoaderFactories?

              Should be fine.

               

              So to get the tests straight.

              Why the three level hierarchy depth?

              Who creates requirements and who provides the capabilities?

              Why the re-export on module and package?

               

              My first feeling is that this could be made simpler -- more straight fwd; no grandparent, parent, etc.

               

              The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.

              What happens if you don't do this?

              • 4. Re: Benchmarking classloaders
                bosschaert

                FYI the OSGi benchmarks that I wrote work on the following three platforms:

                  MC-based JBoss OSGi (-Dframework=jbossmc)

                  MSC / jboss-modules based JBoss OSGi (-Dframework=jbmsc)

                  Felix (-Dframework=felix)

                 

                There are various populations available (and its trivial to add more). E.g. you can run the bundle tests with 750 bundles for all three framework with the following command lines:

                jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=jbossmc

                jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=jbmsc

                jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=felix

                 

                A number of populations for both bundle tests and service tests are run mightly in this hudson job: http://jbmuc.dyndns.org:8280/hudson/job/jbosgi-performance-matrix

                 

                For the latest results, see here: http://jbmuc.dyndns.org:8280/hudson/job/jbosgi-performance-matrix/ws/testsuite/performance/target/performance-report.html

                • 5. Re: Benchmarking classloaders
                  kabirkhan

                  Ales Justin wrote:

                   

                  Ales, can you take a look at how I am setting up the VFSClassLoaderFactories?

                  Should be fine.

                   

                  So to get the tests straight.

                  Why the three level hierarchy depth?

                  The three levels was inspired by David's stuff, as I understood it a load of one class triggers load of some others. It can certainly be made simpler though, so I can do a one-level set tomorrow.

                  Ales Justin wrote:

                  Who creates requirements and who provides the capabilities?

                  Why the re-export on module and package?

                  The requirements and capabilities are set up in the individial tests. I'm  not sure about the re-export, is it needed when Impl (loaderA) loads its superclass AbstractImpl (loaderB) which in turn brings in its interface (loaderC)?

                  The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.

                  What happens if you don't do this?

                  I get an exception, IIRC NoClassDefFoundError when trying to load the superclass which comes from a loader that only exports the module and not also the packages. But maybe there is another setting to get around this?

                  • 6. Re: Benchmarking classloaders
                    bosschaert

                    Kabir Khan wrote:

                     

                    Ales Justin wrote:

                     

                    Why the three level hierarchy depth?

                    The three levels was inspired by David's stuff, as I understood it a load of one class triggers load of some others. It can certainly be made simpler though, so I can do a one-level set tomorrow.

                    I created the bundle bench mark test model to follow typical OSGi deployments. Multiple levels are very common in that case, for instance:

                     

                    A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.

                    • 7. Re: Benchmarking classloaders
                      kabirkhan

                      David Bosschaert wrote:

                      I created the bundle bench mark test model to follow typical OSGi deployments. Multiple levels are very common in that case, for instance:

                       

                      A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.

                      I understand and agree, but I think for this Ales wants a bare-bones comparison between exact classloading and the full visibility model, in this case it will probably be clearer with just one level of classes. I'm working on this now, and should hopefully have something in time for today's meeting.

                      • 8. Re: Benchmarking classloaders
                        alesj
                        A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.

                        This doesn't sound like it would need 3-level hierarchy, but rather just 3 properly wired bundles at the same level.

                        • 9. Re: Benchmarking classloaders
                          kabirkhan

                          The new simpler scenario has a 100 jars with 10 packages and 15 classes per package, one loader per jar.

                           

                          The tests are called SiblingVFSXXXBenchmarkTestCase and they run as follows:

                           

                          Use Loader1 to load all its own classes and those from Loader2

                          Use Loader2 to load all its own classes and those from Loader3

                          ...

                           

                          For the exact classloading setups I make Loader 1 import the module/packages exported by Loader2 etc.

                           

                          I run the tests individually using e.g.

                          mvn install -Dtest=SiblingVFSImportPackageLoaderBenchmarkTestCase

                           

                          Running each test individually, I get these average results over 7 runs

                           

                           

                           


                          Deploy (ms)Load Classes (ms)
                          Big ball of mud14394485
                          Package13214880
                          Module14614874

                           

                          The code lives in svn under https://svn.jboss.org/repos/jbossas/projects/cl-benchmark/trunk

                           

                          Each family of tests uses its own AbstractTestSetCreator implementation which generates jars under their own directory, e.g. ThreeDeepTestSetCreator and SiblingTestCreator. The tests currently

                          • 10. Re: Benchmarking classloaders
                            kabirkhan

                            The times with the latest jboss-cl trunk:

                             

                             


                            Deploy (ms)First Load (ms)Load Cached
                            Big ball of mud97766912
                            Package109370860
                            Module120571060

                             

                            • Deploy is the time taken to install the classloader
                            • First Load is the time taken to load the class first time
                            • Load Cached is the time taken to load the class once it is already loaded

                            In all cases the class comes from another loader than the initiating loader