Benchmarking classloaders

[kabirkhan]

For AS 6 we might want to look at setting up imports and exports for the classloaders, which might be faster than using the current ImportAll/Big-ball-of-mud setting. However, that sounds like a very big task, so I we should benchmark this in isolation to see if imports/exports will actually be faster. I have something simple working for ImportAll, but need to think a bit about how to do this with imports and exports.

[bosschaert]

Hi Kabir,

I just implemented something like this for the OSGi performance benchmarks. Here's a quick description of the bundle dependency topology that it creates:

5 (Versioned) Common Bundles
   - Exports org.jboss.osgi.test.common;version=x
5 Numbered (but not versioned) Util Bundles
   - Imports org.jboss.osgi.test.common
   - Exports org.jboss.osgi.test.util[x];uses="org.jboss.osgi.test.common"
5 Versioned Interfaces Bundles
   - Exports org.jboss.osgi.test.versioned;version=x
5 Versioned Impl Bundles
   - Imports org.jboss.osgi.test.common;version=[x,x]
   - Imports org.jboss.osgi.test.versioned;version=[x,x]
   - Imports org.jboss.osgi.test.util[x]
   - Exports org.jboss.osgi.test.versioned.impl;version=x;uses=org.jboss.osgi.test.util[x]
a large number of test bundles (number configurable)
   - Imports org.jboss.osgi.test.common;version=[x,x]
   - Imports org.jboss.osgi.test.versioned;version=[x,x]
   - Imports org.jboss.osgi.test.versioned.impl;version=[x,x]

Where x is a number [1..5].

Each test bundle loads a class of each of its 3 dependency packages in its activator. This also triggers an indirect load on the Util[x] class.

It's available on master in jbosgi on github: BundleInstallAndStartBenchmark.java

You might be able to reuse some of this...

David

[kabirkhan]

I have got some basic benchmarks working. Instead of using AS classes, I am now generating my own using Chiba's ClassFileWriter to have better control over what exists where. The classes are only generated if the target/generated-classes and target/generated-jars directories don't exist already. You tell it how many jars to create, how many packages per jar and how many classes per package. If you change these numbers, be sure to delete the target/generated-* directories. So, if 2 is selected for each of these it then creates the following jars:

interface0.jar:
org.jboss.test.interface0.pkg0.Interface0.class
org.jboss.test.interface0.pkg0.Interface1.class
org.jboss.test.interface0.pkg1.Interface0.class
org.jboss.test.interface0.pkg1.Interface1.class
 
 
interface1.jar:
org.jboss.test.interface1.pkg0.Interface0.class
org.jboss.test.interface1.pkg0.Interface1.class
org.jboss.test.interface1.pkg1.Interface0.class
org.jboss.test.interface1.pkg1.Interface1.class
 
 
abstract0.jar:
org.jboss.test.abstract0.pkg0.Abstract0.class
org.jboss.test.abstract0.pkg0.Abstract1.class
org.jboss.test.abstract0.pkg1.Abstract0.class
org.jboss.test.abstract0.pkg1.Abstract1.class
 
 
abstract1.jar:
org.jboss.test.abstract1.pkg0.Abstract0.class
org.jboss.test.abstract1.pkg0.Abstract1.class
org.jboss.test.abstract1.pkg1.Abstract0.class
org.jboss.test.abstract1.pkg1.Abstract1.class
 
 
impl0.jar:
org.jboss.test.impl0.pkg0.Impl0.class
org.jboss.test.impl0.pkg0.Impl1.class
org.jboss.test.impl0.pkg1.Impl0.class
org.jboss.test.impl0.pkg1.Impl1.class
 
 
impl1.jar:
org.jboss.test.impl1.pkg0.Impl0.class
org.jboss.test.impl1.pkg0.Impl1.class
org.jboss.test.impl1.pkg1.Impl0.class
org.jboss.test.impl1.pkg1.Impl1.class

An example of the expected inheritance hierarchies is:

class org.jboss.test.impl1.pkg1.Impl1 extends org.jboss.test.abstract1.pkg1.Abstract1{}
class org.jboss.test.abstract1.pkg1.Abstract1 implements org.jboss.test.interface1.pkg1.Interface1{}

I then create a classloader for each jar, with different classloading rules per test and try to load the Impl classes from the implx.jar loaders, which triggers searches for the superclasses/interfaces in the other classloaders.

Ales, can you take a look at how I am setting up the VFSClassLoaderFactories? I do this in the TestCase classes. The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.

Running this a few times on a set of 50 jars with 10 packages per jar and 10 classes per package, I get the following results.:

Import/Export packages:
Deploying the VFSClassLoaderFactories and creating the loaders: 1862ms 1693ms 1847ms 1725ms 1706ms
Loading the classes: 5008ms 4883ms 4744ms 4893ms 5012ms
 
 
Export/ImportAll:
Deploying the VFSClassLoaderFactories and creating the loaders: 1657ms 1627ms 1659ms 1720ms 1573ms
Loading the classes: 4858ms 4693ms 4796ms 4714ms 4899ms
 
 
Import/Export module:
Deploying the VFSClassLoaderFactories and creating the loaders: 1785ms 1736ms 1766ms 1819ms 1901ms
Loading the classes: 4992ms 5609ms 5510ms 4991ms 5626ms

[alesj]

Ales, can you take a look at how I am setting up the VFSClassLoaderFactories?

Should be fine.

So to get the tests straight.

Why the three level hierarchy depth?

Who creates requirements and who provides the capabilities?

Why the re-export on module and package?

My first feeling is that this could be made simpler -- more straight fwd; no grandparent, parent, etc.

The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.

What happens if you don't do this?

[bosschaert]

FYI the OSGi benchmarks that I wrote work on the following three platforms:

  MC-based JBoss OSGi (-Dframework=jbossmc)

  MSC / jboss-modules based JBoss OSGi (-Dframework=jbmsc)

  Felix (-Dframework=felix)

There are various populations available (and its trivial to add more). E.g. you can run the bundle tests with 750 bundles for all three framework with the following command lines:

jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=jbossmc

jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=jbmsc

jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=felix

A number of populations for both bundle tests and service tests are run mightly in this hudson job: http://jbmuc.dyndns.org:8280/hudson/job/jbosgi-performance-matrix

For the latest results, see here: http://jbmuc.dyndns.org:8280/hudson/job/jbosgi-performance-matrix/ws/testsuite/performance/target/performance-report.html

[kabirkhan]

Ales Justin wrote:

Ales, can you take a look at how I am setting up the VFSClassLoaderFactories?

Should be fine.

So to get the tests straight.

Why the three level hierarchy depth?

The three levels was inspired by David's stuff, as I understood it a load of one class triggers load of some others. It can certainly be made simpler though, so I can do a one-level set tomorrow.

Ales Justin wrote:

Who creates requirements and who provides the capabilities?

Why the re-export on module and package?

The requirements and capabilities are set up in the individial tests. I'm  not sure about the re-export, is it needed when Impl (loaderA) loads its superclass AbstractImpl (loaderB) which in turn brings in its interface (loaderC)?

The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.

What happens if you don't do this?

I get an exception, IIRC NoClassDefFoundError when trying to load the superclass which comes from a loader that only exports the module and not also the packages. But maybe there is another setting to get around this?

[bosschaert]

Kabir Khan wrote:

Ales Justin wrote:

Why the three level hierarchy depth?

The three levels was inspired by David's stuff, as I understood it a load of one class triggers load of some others. It can certainly be made simpler though, so I can do a one-level set tomorrow.

I created the bundle bench mark test model to follow typical OSGi deployments. Multiple levels are very common in that case, for instance:

A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.

[kabirkhan]

David Bosschaert wrote:

I created the bundle bench mark test model to follow typical OSGi deployments. Multiple levels are very common in that case, for instance:

A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.

I understand and agree, but I think for this Ales wants a bare-bones comparison between exact classloading and the full visibility model, in this case it will probably be clearer with just one level of classes. I'm working on this now, and should hopefully have something in time for today's meeting.

[alesj]

A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.

This doesn't sound like it would need 3-level hierarchy, but rather just 3 properly wired bundles at the same level.

[kabirkhan]

The new simpler scenario has a 100 jars with 10 packages and 15 classes per package, one loader per jar.

The tests are called SiblingVFSXXXBenchmarkTestCase and they run as follows:

Use Loader1 to load all its own classes and those from Loader2

Use Loader2 to load all its own classes and those from Loader3

...

For the exact classloading setups I make Loader 1 import the module/packages exported by Loader2 etc.

I run the tests individually using e.g.

mvn install -Dtest=SiblingVFSImportPackageLoaderBenchmarkTestCase

Running each test individually, I get these average results over 7 runs

	Deploy (ms)	Load Classes (ms)
Big ball of mud	1439	4485
Package	1321	4880
Module	1461	4874

The code lives in svn under https://svn.jboss.org/repos/jbossas/projects/cl-benchmark/trunk

Each family of tests uses its own AbstractTestSetCreator implementation which generates jars under their own directory, e.g. ThreeDeepTestSetCreator and SiblingTestCreator. The tests currently

[kabirkhan]

The times with the latest jboss-cl trunk:

	Deploy (ms)	First Load (ms)	Load Cached
Big ball of mud	977	669	12
Package	1093	708	60
Module	1205	710	60

• Deploy is the time taken to install the classloader
• First Load is the time taken to load the class first time
• Load Cached is the time taken to load the class once it is already loaded

In all cases the class comes from another loader than the initiating loader