Using the JBoss AS Clustering Testsuite

This document is meant to provide AS developers with background information on how to use the clustering portion of the AS testsuite. The document is targeted at the current AS trunk, i.e. for the JBoss 5 series.  The clustering testsuite has been considerably reorganized for AS 5, so much of this information does not pertain to the 3.x or 4.x branches.

Quick start -- Running the Testsuite

From the root of the testsuite module:

./build.sh

and then:

./build.sh tests-clustering-all-stacks

The first compiles tests, builds tests deployments, etc.  It does not need to be repeated between runs if you haven't changed anything.

The second command will run all the non-EJB3 clustering tests; repeating many with different JGroups stacks (currently a UDP stack and a TCP stack.) If you are not interested in repeating tests with different stacks and just want to test the AS default stack, you can just:

./build.sh tests-clustering

All EJB3 tests are run separately as EJB3 is in a completely codebase. To run the EJB3 clustering tests, from the root of the testsuite module in your EJB3 checkout:

ant -f build-test.xml
export JBOSS_HOME=/path/to/jbossas/install
ant -f build-test.xml clustered-tests

The first line builds all the various tests and test jars in the overall EJB3 integration testsuite; the last executes the tests against the AS version indicated by $JBOSS_HOME.

Binding cluster nodes

For most tests, the testsuite launches two JBoss AS instances on the machine running the tests, which then form a cluster. Each AS needs to bind to a different IP address to avoid port conflicts.  By default, one will bind to localhost and one will bind to the machine's default address.  (The testsuite does this by passing a  value when launching each AS.) These defaults do not work in many environments; for example Windows will not route multicast between the loopback interface and other interfaces, so the two nodes will not cluster. If you have this problem, you will need to multi-home your machine and tell the testsuite to bind one AS to one non-loopback address and the other to a second address.

To do this, edit the  file found in the root of the testsuite module. (For EJB3 there is an equivalent file in the root of EJB3's testsuite module.)  Add entries as follows, assuming 192.168.0.10 and 192.168.0.11 are your machine's IP addresses:

node0=192.168.0.10
node1=192.168.0.11

There are other commented out properties in the file that begin with "node0" and "node1".  You do not need to uncomment or set these.

DO NOT check in your changed  file.

Organization of the Clustering tests in the module

There are a number of variations on the standard AS all config that need to be tested in order to provide proper test coverage, i.e. to ensure that functionality that works properly in the standard configuration also works with other supported configurations, or to test features (like FIELD granularity web session replication) that require changes from the standard config to work at all.  During the course of execution of the tests-clustering target, different AS configurations will be launched and relevant tests will be executed against each configuration.  Some tests will be repeated multiple times, others will only be run once, against the relevant configuration.  The set of requirements is fairly complex.

To ease the pain of this complexity, the clustering test classes and related resource files are organized to make it easy to create a few simple ant patternsets that describe the different sets of tests that need to be executed.  Any new tests that are added must be organized accordingly; don't introduce files in random locations and hack them into the build script, expecting others to maintain the script.

High Level Structure

All files should be located underneath one of the following top level directories:

src/main/org/jboss/test/cluster  for .java files

src/resources/cluster for various resource files (e.g. web.xml files to pack into test wars)

Underneath  there are three main groups of subpackages -- the org.jboss.test.cluster.defaultcfg package, the org.jboss.test.cluster.multicfg package, and various other miscellaneous packages.

The org.jboss.test.cluster.defaultcfg package hierarchy is for JUnit test classes that should only be executed in tests using the standard JGroups protocol stack choices that the AS ships with (i.e. UDP-based stacks).  Tests that need to be executed again with different protocol stacks (i.e. TCP-based ones) should not be located here; they go in org.jboss.test.cluster.multicfg.  Every class underneath the  package hierarchy should be a JUnit test class.  No ancillary support files should be stored underneath  -- they should go in one of the miscellaneously named packages.

The org.jboss.test.cluster.multicfg package hierarchy is for JUnit test classes that need to be executed both with the standard JGroups protocol stack choices and with other supported stacks.  These tests will be run multiple times, once with each stack.  Do not place tests in org.jboss.test.cluster.multicfg if there is no benefit to running them with different protocol stacks -- you're just making the testsuite run longer.  Put them in org.jboss.test.cluster.defaultcfg.  Again, no ancillary support files should be stored underneath  -- they should go in one of the miscellaneously named packages.

If you place a JUnit test class outside the org.jboss.test.cluster.defaultcfg or org.jboss.test.cluster.multicfg hierarchies, it will not get executed.

The miscellaneous packages under org.jboss.test.cluster but outside org.jboss.test.cluster.defaultcfg and org.jboss.test.cluster.multicfg should be used for organizing support classes used in tests (e.g. EJB bean classes, mocks, abstract test superclasses).  How to organize things in these packages is more free-form; no hard rules.  Basically, just group things logically, grouping together classes that support a given category of tests or are used in a particular test deployment.

Web Tier Tests

The organization of the web tier tests is a bit more complex. This reflects the greater variety of deployment configurations that need to be tested.  Some tests need to be repeated with total replication and buddy replication, some need mod_jk integration code enabled, some need JBC region-based-marshalling support.  Basically, the following packages exist:

• org.jboss.test.cluster.defaultcfg.simpleweb.test -- Tests not requiring any deployment to the AS should go here.

• org.jboss.test.cluster.(default | multi)cfg.web.test -- simple web tier tests that only need to be run once, with the same configuration as the standard all config.

• For web tier tests that need to be repeated with buddy replication on and off and with REPL_SYNC on and off.

  • org.jboss.test.cluster.(default | multi)cfg.web.field.test -- any tests of FIELD granularity.

  • org.jboss.test.cluster.(default | multi)cfg.web.jk.test -- any test that exercises integration with JK-based load balancers .  When we test REPL_SYNC we also set UseJK to true, so these tests will only be run with REPL_SYNC. (You could also put tests that are unconcerned about UseJK but that require REPL_SYNC in this package.)

Non Web Tier Tests

There are a few subpackages of org.jboss.test.cluster.defaultcfg and org.jboss.test.cluster.multicfg besides those discussed above.  These are basically just to organize the non-web-tier tests a bit; there is no difference between the packages in how often they get executed. The one exception to this is org.jboss.test.cluster.defaultcfg.cache which contains some outdated tests that are currently excluded from the test run.

Ant Scripts Used by Clustering Testsuite

Building and running the testsuite is controlled by the testsuite module's build.xml file.  This file doesn't include much specific clustering-related content; the clustering test functionality is brought in via ant imports of the following files:

imports/server-config.xml --  This file contains macros and targets related to setting up, starting and stopping different AS configurations.  A good understanding of this file is important to understanding the JBoss testsuite overall.  It is not just used by the clustering testsuite. Of particular interest is the server:config section, wherein we declare JVM and system property settings that should be used in the various AS configurations that the testsuite launches. Important point: if a test target you write needs to launch an AS instance, i.e. can't be run within the scope of an existing AS startup.shutdown, you must create a new server config for it. Do not just restart an existing config. Doing so trashes any existing server logs, making analyzing test problems much more difficult.

imports/sections/cluster.xml -- this controls the build of the clustering portion of the testsuite.  Includes tasks to build the various sars/wars/ears/jars that are used by the tests.  If you need to build a new archive for a test or modify the contents of an existing one, this is where you set that up.

imports/configs/tests-clustering-xml -- this contains the targets that actually execute the tests. The patternsets that define what gets executed are also declared here.  The most critical targets are tests-clustering-all-stacks and tests-clustering.

Running Subsets of the Clustering Testsuite

There are a number of convenience targets to help you run subsets of the clustering testsuite:

tests-clustering-http runs the basic (non-FIELD) web clustering tests, with the standard AS settings for REPL_ASYNC and buddy vs. total replication.

tests-clustering-non-http runs the non-web-related clustering tests.

tests-clustering-field runs the FIELD granularity web clustering tests, with the standard AS settings for REPL_ASYNC and buddy vs. total replication.

one-cluster-test runs specified test(s) against the standard AS configuration.  What test to run is specifed using -Dtest=.  The value to pass to -D is an ant fileset expression (this is quite different from the usage of  in the general purpose "one-test" target).  For example, -Dtest=org/jboss/test/defaultcfg/test/*TestCase.class.  Note the use of '/' instead of '.' -- the value identifies files, not java packages!! The '*' wildcard works as well. This target is poorly named, as it lets you run multiple tests.  To run just one test, make the fileset expression more specific and make sure it includes the .class filename extension. So, -Dtest=org/jboss/test/defaultcfg/test/CacheManagerUnitTestCase.class will work; -Dtest=org/jboss/test/defaultcfg/test/CacheManagerUnitTestCase will not.

one-cluster-test-field is the same as above but launches an AS configured to support FIELD granularity web sessions.

one-cluster-test-nostart is the same as above but does not launch the AS instances for you; you do that manually.  Useful if you want to run the same test multiple times without starting/stopping 2 AS instances each time.

Check the tests-clustering.xml file, more convenience targets get added fairly regularly.

Controlling the JGroups Stack

For all of the clustering targets (except ) you can control the stack JGroups uses by passing a system property to ant, with a value of either "tcp" or "udp" (the default). Examples:

./build.sh -Djboss-junit-configuration=tcp tests-clustering

./build.sh -Djboss-junit-configuration=udp tests-clustering-field

Other Tips and Tricks

  Setting up a cluster with localhost addresses

Useful if you have trouble routing multicast between interfaces and you don't want to add a second IP to an external interface. Also useful if you don't want to enable multicast on an external interface.

1- in  add:

ifconfig lo:2 127.0.0.2
ifconfig lo:3 127.0.0.3

2- in  add:

127.0.0.2 localhost2.localdomain lo2
127.0.0.3 localhost3.localdomain lo3

3- run multicast-localhost script:

route add -net 224.0.0.0 netmask 240.0.0.0 dev lo

4- start 1st AS with ""

5- start 2nd AS with ""