Accessing counter from BMUnit

[ibrencic]

I would like to assert in my JUnit test how many times a certain method has executed. For this I create a rule that creates a counter by calling "createCounter()". How can I access the value of this counter from my JUnit test?

[adinn]

Hi Ivan,

This is a specific case of a more general requirement. You create and update some data from a Byteman rule which gets triggered by your app code but you want to be able to access the same data from the test code. The answer is to inject a rule into your test code!

Let's assume your counter rule looks like this

# file myscript.btm
RULE count app calls
CLASS org.my.app.AppClass
Method myMethod
AT ENTRY
IF true
DO incrementCounter("myMethod counter")
ENDRULE

Now, let's say you exercise your using a test in TestClass

package org.my.test
class TestClass
{
  @Test
  @RunWith(BMUnitRunner)
  @BMScript("myscript.btm")
  public void testMyMethod()
  {
    // exercise the app code
    AppClass a = new AppClass();
    a.doWork();
   // how do we retrieve the counter???
    int counterVal = ???
    assert(counterval == 3);
  }
  . . .

The way to do this is to add a method to the test class and inject some rule code into it. Here is the extra rule you need

# file myScript.btm
RULE count app calls
. . .
RULE retrieve counter
CLASS org.my.test.TestClass
METHOD retrieveCounter()
AT ENTRY
IF true
DO return getCounter"myMethod counter")
ENDRULE

and here is what you need to change in the test class

package org.my.test
class TestClass
{
  @Test
  @RunWith(BMUnitRunner)
  @BMScript("myscript.btm")
  public void testMyMethod()
  {
    . . .
    // retrieve the counter
    int counterval = retrieveCounter();
    assert (counterval == 3);
  }
  private int retrieveCounter()
  {
    // ignore the next line -- Byteman will return the real value
    return 0;
  }
}

This technique generalises to a whole load of other interesting cases which are well worth studying. For example, the BMUnit Advanced Fault Injection tutorial (see link on docs page)  injects rendezvous calls into test code and app code, allowing the test program to stop and start several application threads at known points. This is a very valuable technique for testing timing issues in multi-threaded code.

Anyway, I hope the example above is enough to do what you want. If not feel free to ask for more advice.

regards,

Andrew Dinn

[ibrencic]

In case I am testing the same JVM that is running the tests the approach you wrote works for me. But whenever I run a separate target JVM and set "inhibitAgentLoad = true", I get back "0" that is the default value of the retrieveCounter() method. Is this the expected behavior, or do I make some mistake?

[adinn]

Hi Ivan,

Ivan Brencsics wrote:

In case I am testing the same JVM that is running the tests the approach you wrote works for me. But whenever I run a separate target JVM and set "inhibitAgentLoad = true", I get back "0" that is the default value of the retrieveCounter() method. Is this the expected behavior, or do I make some mistake?

Oh yes, of course, the technique I referred to only works when the test program and app are in the same JVM. Sorry, I did not realise you were discussing how to do this in the setup reported in your previous queries.

There are two possible paths towards resolving this. The one which would probably appeal most is actually the one which is not implemented and,even if implemented, would maybe work but only by the skin of its teeth.

There is a long-standing open Byteman feature request to provide a subclass of Helper called DistributedHelper which implements the same API as Helepr (i.e. provides all the documented, standard builtin methods) but across multiple JVMs (see BYTEMAN-191 and also the subordinate issue BYTEMAN-170). This is sort of the solution you want.

I say sort of what you want because it's not quite so simple a fix as it sounds. It's true that an implementation of DistributedHelper would solve the basic problem of sharing Byteman state between several JVMs. If you can arrange to load the agent into both the test JVM (the one running the JUnit test) and the app JVM (the one running the app code being tested) then all you need to do is add HELPER DistributedHelper to the rule which calls incrementCounter("myMethod counter") on the app JVM and to the rule which calls getCounter("myMethod counter") on the test JVM, load each rule into the relevant JVM and then the same counter wil be used for the increment and get operation.

The problem is that those 'ifs' don't chime precisely with how BMUnit works. First off, BMUNit will only load the agent into zero or one JVM Instances. You need to load an agent into two JVM instances. So, you will have to configure at least one JVM install on the command (both might be best). That implies you will have to specify a different port for each JVM -- you cannot use the same host for both or else one of them will get a failure trying to open its listener socket So, for example, you could use foobar.my.org:9091 for the app JVM and foobar.my.org:9092 for the test JVM.

Secondly, how do you get BMUnit to upload the rules to the correct JVM? Well, you can cheat here because Byteman allows you to configure at BMUnit at both class and method level. So, if you place a BMUnitConfig annotation on the class specifying host and port to identify foobar.my.org:9091 then you could upload the app rule using a BMScript annotation on the class. However, when you come to run the test you would need  to add another BMUnitConfig annotation on the method specifying host and port to identify foobar.my.org:9092. BMUnit and attach a BMScript annottaion pointing to a file containing the rule for use in the test JVM. This trick would work when only two JVMs are involved but not for more.

Anyway, that's all a bit academic since, relatively easy as it is to implement DistributedHelper no one has actually got round to doing it yet :-/

So, what is the alternative? Well, if you look in the Byteman source tree you will see a package call DTest (look in contrib/dtest). It is a tool which allows you to instrument individual methods, classes or packages which you want to execute in a remote JVM (which needs the Byteman agent installed into it). When you start your test you use the API provided by DTest to genertae and upload rules to the remote JVM. When you execute the test code in the remote JVM the rules the custom helper posts back a record of what callsoccurred during that run and what arguments were passed in each call. You can then search the trace history and make assertions about what happened to validate the test run.

This package is not integrated with BMUnit. So, you cannot use the BMUnitRunner to drive the test and  upload and unload of the rules. You have to do the latter by making your test code call out to the DTest API. You can drive the test from JUnit so long as you ensure that i) the byteman-dtest jar is in your test classpath and ii)  the remoteJVM is started with the Byteman agent installed and with byteman-dtest.jar in its classpath. Minimal details of how to use dtest are provided in the README file. You will probably need to look at the javadoc for the package (it's all in maven central) to get a full idea of how to use the package. Alternatively,look at the XTS tests it was originally written to support.  Don't believe the comment inthe README about this being prototype code -- it is in use in more than one project inside and outside Red Hat.