I decided to use a global flag variable (visible to all rules) to indicate when the message handler has been called. This flag can be set from any thread. I then added another rule, executed on the main test thread, which checks that flag at an appropriate time and raises an exception. The exception can then be handled to call Assert.fail() and fail the JUnit test.
Yes, you have discovered a very useful trick: injecting rules into i) methods which are called by the app code under test and ii) a method implemented by the test thread provides a way of establishing a sideband communication or synchronization channel between test and app code.
A good example of how this can be used for synchronization is provided in the advanced Byteman fault injection tutorial. It injects rules which call rendezous at carefully selected points in the app code and also into a rendezvous method called by the test. The latter accepts a String identifying which rendezvous to meet at. Other rules are provided which allow a rendezvous to be enabled and disabled. That allows the test thread to stop and start application threads at defined points during their execution, ensuring they proceed through timing windows in known orders.
Of course, the existing Byteman primitives are quite limited as regards the sort of information channels they can open up. At present they can only really be used to communicate flag settings or counter values. However, you can easily add your own custom helpers to allow arbitrary objects to be transferred between test and app code. Indeed, I have thought about the idea of adding an arbitrary label method to the helper i.e.
void label(Object label, Object target)
Object labelTarget(Object label)
to enable arbitrary objects to be saved for later retrieval. This can be implemented simply by adding <label, target> to a static hash table in the Helper class. label would allow you to use either a global label (e.g. "foo") or an application instance (e.g. a FileStream) to tag an object encountered during execution (a null target can be used to clear the hash table entry). The test thread can use labelTarget() to retrieve a labelled instance and ensure that it matches up to expectations. A list version might also be useful to build up collections:
void labelList(Object label, Object target)
void clearLabelList(Object label)
List<Object> labelListTarget(Object label)
Alternatively, you could use a combination of a counter and label to list multiple values
label(o.toString() + incrementCounter(o), target)