10 Replies Latest reply on Aug 3, 2010 5:40 AM by kabirkhan

Benchmarking classloaders

kabirkhan Jul 19, 2010 7:01 AM

For AS 6 we might want to look at setting up imports and exports for the classloaders, which might be faster than using the current ImportAll/Big-ball-of-mud setting. However, that sounds like a very big task, so I we should benchmark this in isolation to see if imports/exports will actually be faster. I have something simple working for ImportAll, but need to think a bit about how to do this with imports and exports.

1. Re: Benchmarking classloaders

bosschaert Jul 19, 2010 7:20 AM (in response to kabirkhan)

Hi Kabir,

I just implemented something like this for the OSGi performance benchmarks. Here's a quick description of the bundle dependency topology that it creates:
5 (Versioned) Common Bundles
   - Exports org.jboss.osgi.test.common;version=x
5 Numbered (but not versioned) Util Bundles
   - Imports org.jboss.osgi.test.common
   - Exports org.jboss.osgi.test.util[x];uses="org.jboss.osgi.test.common"
5 Versioned Interfaces Bundles
   - Exports org.jboss.osgi.test.versioned;version=x
5 Versioned Impl Bundles
   - Imports org.jboss.osgi.test.common;version=[x,x]
   - Imports org.jboss.osgi.test.versioned;version=[x,x]
   - Imports org.jboss.osgi.test.util[x]
   - Exports org.jboss.osgi.test.versioned.impl;version=x;uses=org.jboss.osgi.test.util[x]
a large number of test bundles (number configurable)
   - Imports org.jboss.osgi.test.common;version=[x,x]
   - Imports org.jboss.osgi.test.versioned;version=[x,x]
   - Imports org.jboss.osgi.test.versioned.impl;version=[x,x]
Where x is a number [1..5].

Each test bundle loads a class of each of its 3 dependency packages in its activator. This also triggers an indirect load on the Util[x] class.

It's available on master in jbosgi on github: BundleInstallAndStartBenchmark.java

You might be able to reuse some of this...

David
Actions

2. Re: Benchmarking classloaders

kabirkhan Jul 21, 2010 1:03 PM (in response to bosschaert)

I have got some basic benchmarks working. Instead of using AS classes, I am now generating my own using Chiba's ClassFileWriter to have better control over what exists where. The classes are only generated if the target/generated-classes and target/generated-jars directories don't exist already. You tell it how many jars to create, how many packages per jar and how many classes per package. If you change these numbers, be sure to delete the target/generated-* directories. So, if 2 is selected for each of these it then creates the following jars:

interface0.jar:
org.jboss.test.interface0.pkg0.Interface0.class
org.jboss.test.interface0.pkg0.Interface1.class
org.jboss.test.interface0.pkg1.Interface0.class
org.jboss.test.interface0.pkg1.Interface1.class
 
 
interface1.jar:
org.jboss.test.interface1.pkg0.Interface0.class
org.jboss.test.interface1.pkg0.Interface1.class
org.jboss.test.interface1.pkg1.Interface0.class
org.jboss.test.interface1.pkg1.Interface1.class
 
 
abstract0.jar:
org.jboss.test.abstract0.pkg0.Abstract0.class
org.jboss.test.abstract0.pkg0.Abstract1.class
org.jboss.test.abstract0.pkg1.Abstract0.class
org.jboss.test.abstract0.pkg1.Abstract1.class
 
 
abstract1.jar:
org.jboss.test.abstract1.pkg0.Abstract0.class
org.jboss.test.abstract1.pkg0.Abstract1.class
org.jboss.test.abstract1.pkg1.Abstract0.class
org.jboss.test.abstract1.pkg1.Abstract1.class
 
 
impl0.jar:
org.jboss.test.impl0.pkg0.Impl0.class
org.jboss.test.impl0.pkg0.Impl1.class
org.jboss.test.impl0.pkg1.Impl0.class
org.jboss.test.impl0.pkg1.Impl1.class
 
 
impl1.jar:
org.jboss.test.impl1.pkg0.Impl0.class
org.jboss.test.impl1.pkg0.Impl1.class
org.jboss.test.impl1.pkg1.Impl0.class
org.jboss.test.impl1.pkg1.Impl1.class

An example of the expected inheritance hierarchies is:

class org.jboss.test.impl1.pkg1.Impl1 extends org.jboss.test.abstract1.pkg1.Abstract1{}
class org.jboss.test.abstract1.pkg1.Abstract1 implements org.jboss.test.interface1.pkg1.Interface1{}

I then create a classloader for each jar, with different classloading rules per test and try to load the Impl classes from the implx.jar loaders, which triggers searches for the superclasses/interfaces in the other classloaders.

Ales, can you take a look at how I am setting up the VFSClassLoaderFactories? I do this in the TestCase classes. The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.

Running this a few times on a set of 50 jars with 10 packages per jar and 10 classes per package, I get the following results.:

Import/Export packages:
Deploying the VFSClassLoaderFactories and creating the loaders: 1862ms 1693ms 1847ms 1725ms 1706ms
Loading the classes: 5008ms 4883ms 4744ms 4893ms 5012ms
 
 
Export/ImportAll:
Deploying the VFSClassLoaderFactories and creating the loaders: 1657ms 1627ms 1659ms 1720ms 1573ms
Loading the classes: 4858ms 4693ms 4796ms 4714ms 4899ms
 
 
Import/Export module:
Deploying the VFSClassLoaderFactories and creating the loaders: 1785ms 1736ms 1766ms 1819ms 1901ms
Loading the classes: 4992ms 5609ms 5510ms 4991ms 5626ms

3. Re: Benchmarking classloaders

alesj Jul 22, 2010 10:40 AM (in response to kabirkhan)

Ales, can you take a look at how I am setting up the VFSClassLoaderFactories?
Should be fine.

So to get the tests straight.
Why the three level hierarchy depth?
Who creates requirements and who provides the capabilities?
Why the re-export on module and package?

My first feeling is that this could be made simpler -- more straight fwd; no grandparent, parent, etc.

The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.
What happens if you don't do this?
Actions
4. Re: Benchmarking classloaders

bosschaert Jul 22, 2010 10:53 AM (in response to bosschaert)

FYI the OSGi benchmarks that I wrote work on the following three platforms:
MC-based JBoss OSGi (-Dframework=jbossmc)
MSC / jboss-modules based JBoss OSGi (-Dframework=jbmsc)
Felix (-Dframework=felix)

There are various populations available (and its trivial to add more). E.g. you can run the bundle tests with 750 bundles for all three framework with the following command lines:
jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=jbossmc
jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=jbmsc
jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=felix

A number of populations for both bundle tests and service tests are run mightly in this hudson job: http://jbmuc.dyndns.org:8280/hudson/job/jbosgi-performance-matrix

For the latest results, see here: http://jbmuc.dyndns.org:8280/hudson/job/jbosgi-performance-matrix/ws/testsuite/performance/target/performance-report.html
Actions
5. Re: Benchmarking classloaders

kabirkhan Jul 22, 2010 3:11 PM (in response to alesj)

Ales Justin wrote:

Ales, can you take a look at how I am setting up the VFSClassLoaderFactories?
Should be fine.

So to get the tests straight.
Why the three level hierarchy depth?
The three levels was inspired by David's stuff, as I understood it a load of one class triggers load of some others. It can certainly be made simpler though, so I can do a one-level set tomorrow.
Ales Justin wrote:
Who creates requirements and who provides the capabilities?
Why the re-export on module and package?
The requirements and capabilities are set up in the individial tests. I'm not sure about the re-export, is it needed when Impl (loaderA) loads its superclass AbstractImpl (loaderB) which in turn brings in its interface (loaderC)?
The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.
What happens if you don't do this?
I get an exception, IIRC NoClassDefFoundError when trying to load the superclass which comes from a loader that only exports the module and not also the packages. But maybe there is another setting to get around this?
Actions
6. Re: Benchmarking classloaders

bosschaert Jul 23, 2010 2:51 AM (in response to kabirkhan)

Kabir Khan wrote:

Ales Justin wrote:

Why the three level hierarchy depth?
The three levels was inspired by David's stuff, as I understood it a load of one class triggers load of some others. It can certainly be made simpler though, so I can do a one-level set tomorrow.
I created the bundle bench mark test model to follow typical OSGi deployments. Multiple levels are very common in that case, for instance:

A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.
Actions
7. Re: Benchmarking classloaders

kabirkhan Jul 23, 2010 5:38 AM (in response to bosschaert)

David Bosschaert wrote:
I created the bundle bench mark test model to follow typical OSGi deployments. Multiple levels are very common in that case, for instance:

A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.
I understand and agree, but I think for this Ales wants a bare-bones comparison between exact classloading and the full visibility model, in this case it will probably be clearer with just one level of classes. I'm working on this now, and should hopefully have something in time for today's meeting.
Actions
8. Re: Benchmarking classloaders

alesj Jul 23, 2010 9:29 AM (in response to bosschaert)

A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.
This doesn't sound like it would need 3-level hierarchy, but rather just 3 properly wired bundles at the same level.
Actions
9. Re: Benchmarking classloaders

kabirkhan Jul 23, 2010 9:40 AM (in response to kabirkhan)

The new simpler scenario has a 100 jars with 10 packages and 15 classes per package, one loader per jar.

The tests are called SiblingVFSXXXBenchmarkTestCase and they run as follows:

Use Loader1 to load all its own classes and those from Loader2
Use Loader2 to load all its own classes and those from Loader3
...

For the exact classloading setups I make Loader 1 import the module/packages exported by Loader2 etc.

I run the tests individually using e.g.
mvn install -Dtest=SiblingVFSImportPackageLoaderBenchmarkTestCase

Running each test individually, I get these average results over 7 runs

Deploy (ms) Load Classes (ms)
Big ball of mud 1439 4485
Package 1321 4880
Module 1461 4874

The code lives in svn under https://svn.jboss.org/repos/jbossas/projects/cl-benchmark/trunk

Each family of tests uses its own AbstractTestSetCreator implementation which generates jars under their own directory, e.g. ThreeDeepTestSetCreator and SiblingTestCreator. The tests currently
Actions
10. Re: Benchmarking classloaders

kabirkhan Aug 3, 2010 5:40 AM (in response to kabirkhan)
The times with the latest jboss-cl trunk:

Deploy (ms) First Load (ms) Load Cached
Big ball of mud 977 669 12
Package 1093 708 60
Module 1205 710 60

Deploy is the time taken to install the classloader
First Load is the time taken to load the class first time
Load Cached is the time taken to load the class once it is already loaded
In all cases the class comes from another loader than the initiating loader
Actions

	Deploy (ms)	Load Classes (ms)
Big ball of mud	1439	4485
Package	1321	4880
Module	1461	4874

	Deploy (ms)	First Load (ms)	Load Cached
Big ball of mud	977	669	12
Package	1093	708	60
Module	1205	710	60

Go to original post