-
1. Re: Testing multi-threaded code with Byteman
Hi Geoffrey :-)
Thanks for asking about this on the forum.
Geoffrey De Smet wrote:
Here's the scenario I got (pseudo code):@Test(timeout = 600000)
public void testSomething() {
// 1) initialize a backgroundThread
backgroundThread = new BackgroundThread();
backgroundThread.start();
// 2) Wait for the backgroundThread to finish starting up (so until
it blocks)
waitFor(backgroundThread.ready());
// then, in the mainThread, add 5 elements to the queue
queue.add(0);
queue.add(1);
queue.add(2);
queue.add(3);
queue.add(4);
// 3) Wait for the backgroundThread to consume all 5 elements
waitFor(queue.remove(4));
// then, in the mainThread, add 3 more elements to the queue
queue.add(5);
queue.add(6);
queue.add(7);
// 4) Wait for the backgroundThread to consume those 3 elements too
waitFor(queue.remove(7));
// then, in the mainThread, tell the backgroundThread to shutdown
backgroundThread.shutdown()
// 5) Wait for the backgroundThread to shutdown
backgroundThread.join();
}
How would something like that look like in Byteman?
The classic way to do this is to inject code into the test thread and the background thread so that they synchronize at the point where you want to suspend and resume execution. I'll talk you thorug this as a worked example because it is probaby going to help lots of other people too. If you want a more full example read both Byteman tutorial, especially the second one on Fault Injection. You can find links to them on the Byteman docs page (you will also find a link to the latest programmer's guide).
Let's use your first wait point to provide an example of the basic technique:
Geoffrey De Smet wrote:
@Test(timeout = 600000)
public void testSomething() {
// 1) initialize a backgroundThread
backgroundThread = new BackgroundThread();
backgroundThread.start();
// 2) Wait for the backgroundThread to finish starting up (so until
it blocks)
waitFor(backgroundThread.ready());
I'm assuming that your background thread actually calls a method BackgroundThread.ready() before it does any work (it's not really critical to how the thing works). You can use a Byteman rule to create a rendezvous object when the thread is created and two more rules to make the test thread and background thread synchronize using that rendezvous when ready() gets called. We will use the background thread instance to identify the rendezvous since that is a data value accessible to both threads (we could use any object they can both share like a specific String or an int but let's stick with the thread).
Here is a Byteman rule to create the rendezvous:
RULE create repeatable rendezvous for 2 parties
CLASS BackgroundThread
METHOD <init>
AT EXIT
IF true
DO createRendezvous($0, 2, true)
ENDRULE
The Byteman rule identifies where in the code base to inject the code (CLASS/INTERFACE, METHOD and AT clauses), a Java boolean expression (IF clause) which guards execution of the one or more Java expressions which constitute the rule's actions (DO clause). The values appearing in some of these clauses can be more or less specific depending upon how precisely you define things so I will go through them one by one.
The class is specified without a package qualifier so the rule will apply to any class with unqualified name BackgroundThread (in any classloader).
The method uses the special (internal) name for a constructor <init>. Similarly, you can use <clinit> to inject into the class's static initialization code. For other methods you use the normal name e.g. add, append for LinkedList etc.
The method clause does not specify a signature so the rule applies to all constructors. If instead you wrote METHOD <init>(String) then the code would only be injected into the constructor which took a single String arg. n.b. if no such constructor exists in some candidate class called BackgroundThread then Byteman will ignore the rule -- it might apply to some other class with that name loaded in some other classloader.
The condition here is true because we always want to create the rendezvous. However, it could refer to data in the injection context e.g. if the constructor took a String argument name we could use IF $name.equals("geoffrey").
The action is a single Java expression which is a call to a Byteman built-in method -- built-in calls look like function calls but they are actually method invocations on an instance of a POJO class provided by Byteman called Helper. Helper.createRendezvous does what it says on the tin i.e it creates a rendezvous structure -- albeit that's an instance which is managed by the Byteman engine itself. If you want to know precisely what a Rendezvous does look at the code for class Helper. Meanwhile I'll explain the basic operation.
By the by, your conditions and actions don't have to stick to built-in methods. You can actually use any available Java method (i.e. in scope static calls or calls to instance methods of in scope objects) plus the usual Java operators to build expressions appearing in the DO clause. But for what you want Byteman provides some very useful and easy to use built-ins so we will stick to them for now. I'll go through the call arguments so you can understand what this call does.
The first argument, $0, identifies the target of the <init> method i.e. the newly created BackgroundThread. This object is used as an identifier to label the rendezvous allowing you to refer to it in other rules (of course, mentioning $0 is only valid for constructors and instance methods -- you will get a type error id if you inject a rule into a static method which tries to use $0). If you compile your code with -g you can also refer to the target instance as $this. Similarly, you can always refer to call method parameters using $1, $2 etc but you can argument names like $name etc if you compile with -g. Note that Byteman knows the type of $0 is BackgroundThread which is fine because createRendezvous accepts an Object as its first argument. It will complain if your rule does not actually type check against the types in scope at the injection point.
The 2nd int argument is a count of how many threads need to arrive at the rendezvous before anyone can leave. In this case we use 2 because we just want the test thread and the background thread to meet up.
The 3rd boolean argument says whether this rendezvous gets deleted after the rendezvous occurs (false) or whether it is reset to allow repeated rendezvous (true). We want the latter as your scenario requires several synchronizations.
So, if we run the test with this rule injected into the code base (I'll come to how we do that later) then when the test thread creates the background thread Byteman will set up a rendezvous structure that we can use in the next rules.
Ok, what about getting the test thread and background thread to actually rendezvous:
RULE make background thread rendezvous at ready
CLASS BackgroundThread
METHOD ready()
AT ENTRY
IF isRendezvous($this, 2)
DO rendezvous($this);
rendezvous($this)
ENDRULE
This rule is injected into BackgroundThread.ready() at the entry point. The injected condition calls isRendezvous to check that a rendezvous labelled with the current thread and with participant thread count 2 actually exists and, if so, enters the first rendezvous (yes, by calling the built_in method rendezvous :-). That means the thread cannot continue until a 2nd thread also makes a call to rendezvous. The above rule actually makes the background thread enter the rendezvous again (we will see why this is needed in a bit). After meeting twice the thread finishes executing the rule and simply carries on executing the rest of the ready() code.
Ok, now how about the test code. First let's add an empty method to the test class
private void do_rendezvous(Object key, int count)
{
// do nothing -- Byteman does the heavy lifting here
}
Now we use this rule to inject a rendezvous into this method
RULE make test thread rendezvous
CLASS MyTest
METHOD do_rendezvous(Object, int)
IF isRendezvous($key, $count)
DO rendezvous($key, $count)
ENDRULE
So, now at the point where you had this:
// 2) Wait for the backgroundThread to finish starting up (so until
it blocks)
waitFor(backgroundThread.ready());
// then, in the mainThread, add 5 elements to the queue
queue.add(0);
. . .
We substitute this:
// 2) Wait for the backgroundThread to finish starting up (so until
it blocks)
do_rendezvous(backgroundThread, 2);
// then, in the mainThread, add 5 elements to the queue
queue.add(0);
. . .
Now, when the test thread calls do_rendezvous it will execute the injected call to rendezvous. So, once it returns from this call we know that the background thread will have entered and exited the rendezvous via its first call. The test thread can now do what it wants but the background thread has to go past the second rendezvous before it can do anything i.e. it has been latched. The test thread can call code to set up the list and then rendezvous again, only allowing the background thread to proceed and process the list once it is full of data
// 2) Wait for the backgroundThread to finish starting up (so until
it blocks)
do_rendezvous(backgroundThread, 2);
// then, in the mainThread, add 5 elements to the queue
queue.add(0);
queue.add(1);
queue.add(2);
queue.add(3);
queue.add(4);
// unlatch the background thread
do_rendezvous(backgroundThread, 2);
. . .
Ok, so at this point the test thread has rendezvoused with the background thread twice so the latter is now able to go on and start processing the list.
Next we want to perform another synchronization at the point where the background thread has called remove for the 4th time and then a second synchronization when the queue is full again. This is what we need on the test thread side.
. . .
queue.add(3);
queue.add(4);
// unlatch the background thread
do_rendezvous(backgroundThread, 2);
// latch the background thread after it has processed the current queue
do_rendezvous(backgroundThread, 2);
queue.add(5);
queue.add(6);
queue.add(7);
// unlatch the background thread so it can process the next 3 elements
do_rendezvous(background_thread, 2);
But how do we get the background thread to execute 2 more rendezvous at exactly the right point?
Well, we need to modify the behaviour of the background thread so that it is latched at the rendezvous when it has done 5 remove operations and then restarts when we unlatch it. Now there are various ways to do this, which depend upon how the thread code is written. Let's assume that a single call to Queue.remove happens inside a loop in method BackgroundThread.drainQueue()
RULE stop after 5th queue remove called from BackgroundThread method
CLASS BackgroundThread
METHOD drainQueue
AFTER CALL Queue.remove
IF isRendezvous($this, 2) AND incrementCounter($this) == 5
DO rendezvous($this, 2);
rendezvous($this, 2)
ENDRULE
This rule gets injected into method drainQueue at the point where the call to Queue.remove has just returned. n.b. AND is just syntactic sugar for && (which you can use if you prefer). We use another built-in, increment_counter, to keep count of how many times the rule is triggered. increment_counter usefully creates a zeroed counter if it does not exist. It increments the counter when it is called returning the new values i.e. 1, 2, 3,4, 5 at each successive rule triggering. So, the thread will only enter the pair of rendezvous after the 5th call has returned.
If we wanted to rendezous at both the 5th and 8th remove operation we could simply modify the condition as follows
IF isRendezvous($this, 2) AND (incrementCounter($this) == 5 OR readCounter($this) == 8)
Of course, we would also need to add corresponding calls to do_rendezvous to the test code to make sure we latch and unlatch the thread before adding elements 5, 6 and 7 and after they have been added, respectively.
Finally, how do we make sure the background thread has exited? Well, Byteman provides another built-in structure called a joiner and built-in management methods for it precisely for that purpose.
First, we need to create the joiner when we create the rendezvous by modifying our <init> method rule
RULE create joiner and repeatable rendezvous for 2 parties
CLASS BackgroundThread
METHOD <init>
AT EXIT
IF true
DO createRendezvous($0, 2, true);
createJoin($0, 1)
ENDRULE
Now, we need to make the background thread enlist with the joiner when it is about to exit. The child thread just has to enlist at some point during its lifetime -- let's use Thread.exit to do that.The condition makes sure that only a BackgroundThread ever enlists.
RULE enlist in joiner
CLASS Thread
METHOD exit()
AT ENTRY
IF isJoin($0, 1)
DO joinEnlist($0, 1)
ENDRULE
A joiner keeps track of N enlisted threads and will reject all but the first N enlist calls. In this case the joiner was created with count 1 so it only expects one child thread to enlist. Note that when enlist is called the joiner merely adds the joining thread to a list.
Finally, let's add another method to the test class which allows the test thread to do the actual join:
private void do_join_wait(Object key, int count)
{
// leave it to Byteman
}
and here's the rule we need
RULE inject joinWait call
CLASS BackgroundThread
METHOD do_join_wait(Object, int)
AT ENTRY
IF true
DO joinWait($1, $2)
ENDRULE
We need to modify your pseudo code as follows
// then, in the mainThread, tell the backgroundThread to shutdown
backgroundThread.shutdown()
// 5) Wait for the backgroundThread to shutdown
do_join_wait(backgroundThread, 1);Now built-in joinWait() will only proceed when the joiner has had N joinEnlist requests posted i.e. our test thread will get wedged if the background thread does not actually call exit. If N threads have enlisted then joinWait calls Thread.join() for each enlisted thread. So, the test thread will only return from do_join_wait if the background thread has actually gone into the exit call, called joinEnlist and then gone on to exit,allowing the call to Thread.join() to return.
Ok, so that's all the rules we need to code your example. Now, how do you wire up your test to use them? Well, let's assume you want to use JUnit from maven using the surefire plugin. The simplest way is to use the Byteman JUnit integration package BMUnit which is annotation based (n.b. the 2nd Byteman tutorial explains how to use BMUnit with TestNG instead of JUnit and also how to drive the tests from ant instead of maven).
Let's assume we have put all the above rules into a script file in test/resources/myrules.btm. You need to annotate your test as follows
@RunWith(BMUNitRunner.class)
class MyTest {
@Test(timeout = 600000)
@BMScript("value=myrules", dir="target/test-classes")
public void testSomething() {
// 1) initialize a backgroundThread
. . .
The first annotation tells JUnit to use the BMUNit test runner when executing the tests in MyClass. BMUNitRunner is a subclass of JUnit4Runner which does everything that JUnit4Runner does but also knows how to install Byteman into the test JVM and ensure that Byteman rules are injected into the code base before a test is run and removed after the test has completed. The second annotation can be placed either on the class or on a specific test method. It says here are some rules to load before testing and unload after testing. Obviously the different annotation scopes determine whether the rules are injected for all test methods in the class or just for the specific annotated method. Note that maven copies the script file in test/resources to target/test-classes which is why you need to specify the dir="..." attribute.
Of course, to make this work you need to configure maven so it can find the Byteman code. That's also relatively simple to do. You just need to add the 4 byteman jars as test dependencies.
<dependencies>
. . .
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.2</version>
</dependency>
<dependency>
<groupId>org.jboss.byteman</groupId>
<artifactId>byteman</artifactId>
<scope>test</scope>
<version>${byteman.version}</version>
</dependency>
<dependency>
<groupId>org.jboss.byteman</groupId>
<artifactId>byteman-submit</artifactId>
<scope>test</scope>
<version>${byteman.version}</version>
</dependency>
<dependency>
<groupId>org.jboss.byteman</groupId>
<artifactId>byteman-install</artifactId>
<scope>test</scope>
<version>${byteman.version}</version>
</dependency>
<dependency>
<groupId>org.jboss.byteman</groupId>
<artifactId>byteman-bmunit</artifactId>
<scope>test</scope>
<version>${byteman.version}</version>
</dependency>
<dependency>
<groupId>com.sun</groupId>
<artifactId>tools</artifactId>
<version>1.6</version>
<scope>system</scope>
<systemPath>${tools.jar}</systemPath>
</dependency>
</dependencies>
I recommend you use the latest Byteman version (2.1.4.1). Note that you must have tools.jar in your classpath since it conatains the smarts which allows Bytemans Java agent to be loaded into the test JVM. Yes, that means BMUNit will not work unless you have a full JDK installed (as opposed to a JRE).
You are also probably best adding the following boiler-plate to your surefire plugin config
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.4.3</version>
<configuration>
<includes>
<include>...</include>
</includes>
<!-- make sure maven puts the byteman jar in the classpath rather than in a manifest jar -->
<useManifestOnlyJar>false</useManifestOnlyJar>
<forkMode>once</forkMode>
<parallel>false</parallel>
<!-- ensure we don't inherit a byteman jar from any env settings -->
<environmentVariables>
<BYTEMAN_HOME></BYTEMAN_HOME>
</environmentVariables>
<systemProperties>
<property>
<name>org.jboss.byteman.home</name>
<value></value>
</property>
</systemProperties>
</configuration>
</plugin>
The explanation for the settings here is as follows:
parallel -- you don't often want multiple tests to run in parallel because the rules they use might interfere with each other. Imagine two tests use a counter identified by string "foo" to track calls to a specific method. If they run in parallel they might upset each other's counts.
useManifestOnlyJar -- BMUnit has to process the Byteman jar specially (it contains a Java agent program which BMUNit has to load into the JVM using the agent management API in tools.jar) so it needs to be able to identify it from the list of jars in the classpath. If maven uses a manifest jar then the classpath does nto idenitfy the Byteman jar.
forkmode -- the Byteman java agent is automatically loaded into the JVM by BMUnit. That does not cost a lot but if you fork multiple JVMs then this will slow down your tests by requiring it to be loaded repeatedly.
env and sys properties -- BMUnit can and will use a locally downloaded copy of the Byteman agent code which may mean that the byteman jar version is not compatible with the value specified in the pom. Erasing these env variables stops BMUnit looking for a locally downloaded jar.
I hope that is enough to get you started using Byteman for multi-threaded testing. I'll be happy to follow up on any further questions or provide a critique of any code you can provide by way of example. If you do get what you want workign the post a link so others can learn from your experience.
regards,
Andrew Dinn
-
-
3. Re: Testing multi-threaded code with Byteman
Hi Andrew,
That looks pretty impressive, though I'm still getting my head around the specifics. One question though:
It seems like the rendezvous and join rules are fairly boilerplate and would be reused across basically all of the tests in the test class, with only the counter and other details varying per test. Is it possible to factor those common rules into a common BM script and combine it with the test-specific rules in the annotations? Sort of factoring into a more DRY style I guess...
-
4. Re: Re: Testing multi-threaded code with Byteman
Hi John,
John Casey wrote:
That looks pretty impressive, though I'm still getting my head around the specifics.
Thanks very much.
John Casey wrote:
It seems like the rendezvous and join rules are fairly boilerplate and would be reused across basically all of the tests in the test class, with only the counter and other details varying per test. Is it possible to factor those common rules into a common BM script and combine it with the test-specific rules in the annotations? Sort of factoring into a more DRY style I guess...Yes, it is possible to factor the test thread side code & scripts so as to allow reuse. For example, the JBossTS tests make the test class inherit from a super which provides methods to create and delete a rendezvous keyed by a specific String name and to enter a rendezvous on demand. The tests all use a standard rule script to inject the Byteman operations into these inherited methods.
Here is a link to the super class:
and here is the script which provides the corresponding rules:
https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/tests/byteman-scripts/reaper.btm
On the app code side there is less room for sharing. You need a rule specific to the place where you want to stop and start the app code. That tends to be specific to a given test.