-
1. Re: Benchmarking classloaders
bosschaert Jul 19, 2010 7:20 AM (in response to kabirkhan)Hi Kabir,
I just implemented something like this for the OSGi performance benchmarks. Here's a quick description of the bundle dependency topology that it creates:
5 (Versioned) Common Bundles
- Exports org.jboss.osgi.test.common;version=x
5 Numbered (but not versioned) Util Bundles
- Imports org.jboss.osgi.test.common
- Exports org.jboss.osgi.test.util[x];uses="org.jboss.osgi.test.common"
5 Versioned Interfaces Bundles
- Exports org.jboss.osgi.test.versioned;version=x
5 Versioned Impl Bundles
- Imports org.jboss.osgi.test.common;version=[x,x]
- Imports org.jboss.osgi.test.versioned;version=[x,x]
- Imports org.jboss.osgi.test.util[x]
- Exports org.jboss.osgi.test.versioned.impl;version=x;uses=org.jboss.osgi.test.util[x]
a large number of test bundles (number configurable)
- Imports org.jboss.osgi.test.common;version=[x,x]
- Imports org.jboss.osgi.test.versioned;version=[x,x]
- Imports org.jboss.osgi.test.versioned.impl;version=[x,x]Where x is a number [1..5].
Each test bundle loads a class of each of its 3 dependency packages in its activator. This also triggers an indirect load on the Util[x] class.
It's available on master in jbosgi on github: BundleInstallAndStartBenchmark.java
You might be able to reuse some of this...
David
-
2. Re: Benchmarking classloaders
kabirkhan Jul 21, 2010 1:03 PM (in response to bosschaert)I have got some basic benchmarks working. Instead of using AS classes, I am now generating my own using Chiba's ClassFileWriter to have better control over what exists where. The classes are only generated if the target/generated-classes and target/generated-jars directories don't exist already. You tell it how many jars to create, how many packages per jar and how many classes per package. If you change these numbers, be sure to delete the target/generated-* directories. So, if 2 is selected for each of these it then creates the following jars:interface0.jar: org.jboss.test.interface0.pkg0.Interface0.class org.jboss.test.interface0.pkg0.Interface1.class org.jboss.test.interface0.pkg1.Interface0.class org.jboss.test.interface0.pkg1.Interface1.class interface1.jar: org.jboss.test.interface1.pkg0.Interface0.class org.jboss.test.interface1.pkg0.Interface1.class org.jboss.test.interface1.pkg1.Interface0.class org.jboss.test.interface1.pkg1.Interface1.class abstract0.jar: org.jboss.test.abstract0.pkg0.Abstract0.class org.jboss.test.abstract0.pkg0.Abstract1.class org.jboss.test.abstract0.pkg1.Abstract0.class org.jboss.test.abstract0.pkg1.Abstract1.class abstract1.jar: org.jboss.test.abstract1.pkg0.Abstract0.class org.jboss.test.abstract1.pkg0.Abstract1.class org.jboss.test.abstract1.pkg1.Abstract0.class org.jboss.test.abstract1.pkg1.Abstract1.class impl0.jar: org.jboss.test.impl0.pkg0.Impl0.class org.jboss.test.impl0.pkg0.Impl1.class org.jboss.test.impl0.pkg1.Impl0.class org.jboss.test.impl0.pkg1.Impl1.class impl1.jar: org.jboss.test.impl1.pkg0.Impl0.class org.jboss.test.impl1.pkg0.Impl1.class org.jboss.test.impl1.pkg1.Impl0.class org.jboss.test.impl1.pkg1.Impl1.class
An example of the expected inheritance hierarchies is:class org.jboss.test.impl1.pkg1.Impl1 extends org.jboss.test.abstract1.pkg1.Abstract1{} class org.jboss.test.abstract1.pkg1.Abstract1 implements org.jboss.test.interface1.pkg1.Interface1{}
I then create a classloader for each jar, with different classloading rules per test and try to load the Impl classes from the implx.jar loaders, which triggers searches for the superclasses/interfaces in the other classloaders.Ales, can you take a look at how I am setting up the VFSClassLoaderFactories? I do this in the TestCase classes. The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.Running this a few times on a set of 50 jars with 10 packages per jar and 10 classes per package, I get the following results.:Import/Export packages: Deploying the VFSClassLoaderFactories and creating the loaders: 1862ms 1693ms 1847ms 1725ms 1706ms Loading the classes: 5008ms 4883ms 4744ms 4893ms 5012ms Export/ImportAll: Deploying the VFSClassLoaderFactories and creating the loaders: 1657ms 1627ms 1659ms 1720ms 1573ms Loading the classes: 4858ms 4693ms 4796ms 4714ms 4899ms Import/Export module: Deploying the VFSClassLoaderFactories and creating the loaders: 1785ms 1736ms 1766ms 1819ms 1901ms Loading the classes: 4992ms 5609ms 5510ms 4991ms 5626ms
-
3. Re: Benchmarking classloaders
alesj Jul 22, 2010 10:40 AM (in response to kabirkhan)Ales, can you take a look at how I am setting up the VFSClassLoaderFactories?
Should be fine.
So to get the tests straight.
Why the three level hierarchy depth?
Who creates requirements and who provides the capabilities?
Why the re-export on module and package?
My first feeling is that this could be made simpler -- more straight fwd; no grandparent, parent, etc.
The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.
What happens if you don't do this?
-
4. Re: Benchmarking classloaders
bosschaert Jul 22, 2010 10:53 AM (in response to bosschaert)FYI the OSGi benchmarks that I wrote work on the following three platforms:
MC-based JBoss OSGi (-Dframework=jbossmc)
MSC / jboss-modules based JBoss OSGi (-Dframework=jbmsc)
Felix (-Dframework=felix)
There are various populations available (and its trivial to add more). E.g. you can run the bundle tests with 750 bundles for all three framework with the following command lines:
jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=jbossmc
jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=jbmsc
jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=felix
A number of populations for both bundle tests and service tests are run mightly in this hudson job: http://jbmuc.dyndns.org:8280/hudson/job/jbosgi-performance-matrix
For the latest results, see here: http://jbmuc.dyndns.org:8280/hudson/job/jbosgi-performance-matrix/ws/testsuite/performance/target/performance-report.html
-
5. Re: Benchmarking classloaders
kabirkhan Jul 22, 2010 3:11 PM (in response to alesj)Ales Justin wrote:
Ales, can you take a look at how I am setting up the VFSClassLoaderFactories?
Should be fine.
So to get the tests straight.
Why the three level hierarchy depth?
The three levels was inspired by David's stuff, as I understood it a load of one class triggers load of some others. It can certainly be made simpler though, so I can do a one-level set tomorrow.
Ales Justin wrote:
Who creates requirements and who provides the capabilities?
Why the re-export on module and package?
The requirements and capabilities are set up in the individial tests. I'm not sure about the re-export, is it needed when Impl (loaderA) loads its superclass AbstractImpl (loaderB) which in turn brings in its interface (loaderC)?
The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.
What happens if you don't do this?
I get an exception, IIRC NoClassDefFoundError when trying to load the superclass which comes from a loader that only exports the module and not also the packages. But maybe there is another setting to get around this?
-
6. Re: Benchmarking classloaders
bosschaert Jul 23, 2010 2:51 AM (in response to kabirkhan)Kabir Khan wrote:
Ales Justin wrote:
Why the three level hierarchy depth?
The three levels was inspired by David's stuff, as I understood it a load of one class triggers load of some others. It can certainly be made simpler though, so I can do a one-level set tomorrow.
I created the bundle bench mark test model to follow typical OSGi deployments. Multiple levels are very common in that case, for instance:
A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.
-
7. Re: Benchmarking classloaders
kabirkhan Jul 23, 2010 5:38 AM (in response to bosschaert)David Bosschaert wrote:
I created the bundle bench mark test model to follow typical OSGi deployments. Multiple levels are very common in that case, for instance:
A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.
I understand and agree, but I think for this Ales wants a bare-bones comparison between exact classloading and the full visibility model, in this case it will probably be clearer with just one level of classes. I'm working on this now, and should hopefully have something in time for today's meeting.
-
8. Re: Benchmarking classloaders
alesj Jul 23, 2010 9:29 AM (in response to bosschaert)A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.
This doesn't sound like it would need 3-level hierarchy, but rather just 3 properly wired bundles at the same level.
-
9. Re: Benchmarking classloaders
kabirkhan Jul 23, 2010 9:40 AM (in response to kabirkhan)The new simpler scenario has a 100 jars with 10 packages and 15 classes per package, one loader per jar.
The tests are called SiblingVFSXXXBenchmarkTestCase and they run as follows:
Use Loader1 to load all its own classes and those from Loader2
Use Loader2 to load all its own classes and those from Loader3
...
For the exact classloading setups I make Loader 1 import the module/packages exported by Loader2 etc.
I run the tests individually using e.g.
mvn install -Dtest=SiblingVFSImportPackageLoaderBenchmarkTestCase
Running each test individually, I get these average results over 7 runs
Deploy (ms) Load Classes (ms) Big ball of mud 1439 4485 Package 1321 4880 Module 1461 4874 The code lives in svn under https://svn.jboss.org/repos/jbossas/projects/cl-benchmark/trunk
Each family of tests uses its own AbstractTestSetCreator implementation which generates jars under their own directory, e.g. ThreeDeepTestSetCreator and SiblingTestCreator. The tests currently
-
10. Re: Benchmarking classloaders
kabirkhan Aug 3, 2010 5:40 AM (in response to kabirkhan)The times with the latest jboss-cl trunk:
Deploy (ms) First Load (ms) Load Cached Big ball of mud 977 669 12 Package 1093 708 60 Module 1205 710 60 - Deploy is the time taken to install the classloader
- First Load is the time taken to load the class first time
- Load Cached is the time taken to load the class once it is already loaded
In all cases the class comes from another loader than the initiating loader