Cluster Benchmark Harness
What it is:
This is a small project that will run benchmarks on the difference between making local, non-clustered calls and their clustered counterpart calls. The objective is to be able to show the expected performance impact of moving from a non-clustered, co-located environment, to a clustered, distributed environment using JBoss. There are many factors involved in what the actual performance differences will be based on actual applications (such as size of state being replicated, number of nodes in cluster, etc.), but this harness should provide base numbers.
Current benchmark tests are:
Local JNDI lookup vs. HAJNDI lookup for EJB bound during deployment
Local JNDI lookup vs. HAJNDI lookup for object bound using same provider
Local EJB invocation vs. remote EJB invocation
EJB invocation within same classloader repository vs different classloader repositories
How it works:
In the base directory, there is an ant build file. Upon building, two new server nodes will be created, bench1 and bench2, within the specified JBoss server. These nodes will be used to execute the tests. The build also creates some very simple deployments that include Stateless Session Beans. Depending on the tests executed, the build will deploy these beans into the different nodes (bench1 and/or bench2) and run the client tests. Also, depending on the tests run, different configuration files will be copied to nodes, to create the desired test environment.
How to build:
By default, the build file will look in ant.properties file for any properties overrides. If this file does not exist, or if certain properties do not exist, the build file will attempt to use defaults.
The first default that the build file will use is the environment variables. For example, if the JBOSS_HOME property is not defined in the ant.properties file, the build file will then try to see if one was specified in the system&146;s environment variables.
There is a sample ant.properties file provided, but properties will be commented out by default.
How to use:
Although the intention is to fully automate this project, it currently requires some manual configuration. This includes manually editing some configuration files to change the deployment environment, manually starting the test instance nodes of JBoss to run against, and reporting of results is just to standard output.
The first ant target to be run should be the default target &145;all&146;. This will build the test files and their deployment files, two JBoss server tests nodes (bench1 and bench2), as well as configuring these nodes for testing.
You may want to suppress most server logging for these tests to eliminate logging levels as a variable in the tests. Also note that a high logging level may cause a failure in tests that utilize the JBossCache-based HA-JNDI. The issue is described in JBCACHE-279. If the tests generate a NoElementFound exception and temporarily reducing the test threads/iterations resolves the problem, suppress logging for org.jboss.cache to work around the issue.
Note: These tests require a large amount of memory to run. By default, JBoss runs with the JDK&146;s standard 64MB of memory, so will need to change your run.bat (run.sh on unix based OS) to have -Xms128m -Xmx512m or higher set.
JNDI tests:
Make sure the services.properties file in the conf/META-INF/ directory has all the necessary properties for your JNDI test. For example:
java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces java.naming.provider.url=localhost:1099
would be for a test doing a local JNDI lookup.
There are two JNDI tests, run_jndi_test and run_habinding_test.
run_jndi_test performs lookups of an EJB bound during deployment so the EJB is bound in the local naming server while the lookup is performed either through the local naming server or through HA-JNDI, depending on the provider url in the services.properties file.
run_habinding_test performs lookups of a simple object using the same provider for the binding and lookups. The provider is specified via the provider url in the services.properties file.
Start the two JBoss nodes (bench1 and bench2). Then run the appropriate ant target (e.g., ant run_jndi_test) to start the test. The output should look something like:
C:\Projects\JBoss\JBoss-HEAD\jboss-head\testsuite\src\etc\cluster-harness>ant run_jndi_test Buildfile: build.xml run_jndi_test: [java] JNDITestHarness properties [java] -- listing properties -- [java] java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory [java] java.naming.provider.url=localhost:1099 [java] java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces [java] Starting run now with 10 clients and 1000 iterations. [java] Executed 10000 times so far. [java] Ran test with total of 10000 (iterations: 1000, number of clients: 10) in 30 seconds. [java] Average time to make lookup is 29 milliseconds. BUILD SUCCESSFUL Total time: 31 seconds
To change to run using HAJNDI, change the services.properties file to point to the port for the HAJNDI server (this require a re-build) and repeat process.
If you want to change the number of clients (which means threads executing the test) and iterations, set this in the ant.properties file. For example:
test.clients=10 test.iterations=100
would run 10 threads that each performed 100 JNDI lookups.
If you want to see the individual times for each lookup in a test, set the test.print_time property to true. Note that individual lookup times may vary significantly, particularly when running threaded tests.
EJB tests:
To run the EJB local test, do the same as for the JNDI tests, but run the &145;run_local_ejb_test target. Be sure to run &145;ant clean_all most&146; before hand to make sure is clean deployment. As the test runs, should see the following repeated in the console output of the bench1 node instance:
16:30:51,384 INFO [STDOUT] CallerSessionBean:processRequest 16:30:51,384 INFO [STDOUT] SimpleSessionBean:processRequest
and no output within the bench2 console. This is a visual, although somewhat annoying, indicator that the calls being make from the CallerSessionBean to the SimpleSession bean is within the same container, thus a local call.
The output to the cluster-harness console should be similar to the following upon completion:
C:\Projects\JBoss\JBoss-HEAD\jboss-head\testsuite\src\etc\cluster-harness>ant run_local_ejb_test Buildfile: build.xml deploy: run_local_ejb_test: [copy] Copying 1 file to C:\Projects\JBoss\JBoss-HEAD\jboss-head\build\output\jboss-5.0.0alpha\server\bench1\deploy [java] Starting run now with 10 clients and 1000 iterations. [java] Executed 10000 times so far. [java] Ran test with total of 10000 (iterations: 1000, number of clients: 10) in 112 seconds. [java] Average time to make lookup is 111 milliseconds. BUILD SUCCESSFUL Total time: 2 minutes 3 seconds
To run the EJB remote test, first have to change the jndi url to point to bench2 (since will this will be where the CallerSessionBean is deployed and not on bench1). The entry in services.properties file would look like:
java.naming.provider.url=localhost:1199
since this is the local jndi host for bench2. Will then need to do a clean build and then run the &145;run_remote_ejb_test&146; target. This should then deploy the two ear files (simpleClusterSession.ear to bench1 and callerClusterSession.ear to bench2). In the bench1 console, should see the following output repeated for each client call iteration:
17:04:59,649 INFO [STDOUT] SimpleSessionBean:processRequest
}
and nothing else. In the bench2 console, should see the following output repeated for each client call iteration:
17:04:59,659 INFO [STDOUT] CallerSessionBean:processRequest
and nothing else. Once again, this is a visual indicator to show that the CallerSessionBean is being invoked in the bench2 node instance, which then invokes the SimpleSessionBean that is in the bench1 instance, thus proving is a remote invocation.
In the cluster-harness console, should see something like the following upon completion:
C:\Projects\JBoss\JBoss-HEAD\jboss-head\testsuite\src\etc\cluster-harness>ant run_remote_ejb_test Buildfile: build.xml deploy: run_remote_ejb_test: [copy] Copying 1 file to C:\Projects\JBoss\JBoss-HEAD\jboss-head\build\output\jboss-5.0.0alpha\ server\bench2\deploy [delete] Deleting: C:\Projects\JBoss\JBoss-HEAD\jboss-head\build\output\jboss-5.0.0alpha\server\b ench2\deploy\simpleClusterSession.ear [java] Starting run now with 10 clients and 1000 iterations. [java] Executed 10000 times so far. [java] Ran test with total of 10000 (iterations: 1000, number of clients: 10) in 196 seconds. [java] Average time to make lookup is 195 milliseconds. BUILD SUCCESSFUL Total time: 3 minutes 27 seconds
This is executing the EJBTestHarness class. The parameters to running this class are iterations, clients, and jndi url. If you leave the jndi.url off, it will run the tests using local jndi. If you want the CallerSession bean to call out of the container to a SimpleSession bean, you have to specify the jndi url for the container in which the SimpleSession bean is running in (i.e. localhost:1099). Also important to note that will need to use the local jndi port and not the hajndi port to make sure is calling on remote SimpleSession bean. See the run_remote_ejb_test target definition in the ant build.xml file for more details.
Classloader Repository Tests:
Testing invocations between the same loader repository and different loader repositories is simple. To set the results of the same, just run the &145;run_local_ejb_test target&146; as described above. Then to change to use different loader repository, go to the cluster-harness/conf/META-INF/jboss-app.xml and uncomment the loader-repository element. By default, both the SimpleSessionBean deployment and the CallerSessionBean deployment have the loader-repository element commented out, so they both use the default loader repository. By uncommenting this in one, each deployment will have it&146;s own loader repository (default and com.simple:loader=simpleClusterSession.ear).
Issues:
One issue about doing this deployment is how to have build change the jboss-service.xml file in the conf directory so that it uses the ServiceBindingManager. One approach is to just copy a version of the jboss-service.xml with the ServiceBindingManager uncommented and configured into the conf directory. However, this may cause problems between versions as the jboss-service.xml may not be the same.
Can not get &145;start_servers&146; ant target to run on Windows XP. Might be problem running script from ant, but probably just don&146;t have it written correctly.
Need to change so test code so not using system.properties file for dynamic configuration data, such as java.naming.provider.url, since will need want to change this via command line parameters (or figure out a way to change the file easily while running tests so can change from local jndi lookup to hajndi lookup). At the very least, need to change it so don&146;t have to rebuild when make a change to it (as is currently put in the simpleClusterSessionClient.jar&146;s META-INF directory, where it is looked for by the test client and session beans).
In the build file, am deleting the following from the deploy directory:
remoting-service.xml
This is due to the service binding file not correctly binding different addresses for these services (really more of a problem with way remoting service is configured as does not follow the same syntax as other services in regards to bind port).
NOTE: This has only been checked into CVS under jboss-head, so will need to be backported to 3.x or 4.x if desired.
Comments