Hi Jose,
Don't worry about rambling - this all makes sense to me and I am beginning to understand what the problem is that you need to deal with.
Your last idea, which was to try to separate out the agent rules for bootstrap code from the agent rules for non-bootstrap code was actually a step in the right direction. However, you cannot use two separate agents to achieve what you want. If you try to load the Byteman agent twice only one agent will load. That's because the agent has to rely on static variables (don't ask, no really!) so, as they say in the movie Highlander, there can be only one. Even worse, if you add the Byteman jar to both the system and bootstrap classpaths then you risk having some Byteman classes loaded by the system loader and others by the bootstrap loader, causing very mysterious linkage errors and potentially breaking Byteman altogether.
Let's start with what is the root problem here. What you are trying to do involves some different goals that are hard (but, with a little applied trickery, not impossible) to reconcile.
1) You want to inject Byteman rule code into bootstrap path classes -- which, since they belong to the JDK, don't have anything to do with Grizzly
2) You want to inject Byteman rule code into system path classes which belong to Grizzly
3) Under case 1 you want to call out to methods that belong to class TracingHttpServerFilter which need to resolve against Grizzly classes
4) Under case 2 you want to call out to methods that belong to class TracingHttpServerFilter which need to resolve against Grizzly classes
You have a few ways to change the behaviour here. One is to deploy Byteman in the bootstrap path instead of the (default) system path. Another is to insert the jar containing TracingHttpServerFilter and/or the Grizzly jars into the bootstrap path instead of just adding them to the system path.
First, let's consider what happens when you deploy the agent in the system path and the jars containing TracingHttpServerFilter and Grizzly stay in tthe system path. This fails on goal 1. Let's assume your rule wants to inject code into method append of JDK runtime class java.util.List (it's of little or no importance which method it is in realit). The injected code needs to be surrounded with a try catch that captures three potential exceptions that are defined in one of the Byteman packages, EarlyReturnException, ThrowException and RuntimeException. So, these classes are referenced in the modified bytecode that the Byteman agent constructs and installs. When the new version of java.util.List.append() gets called you get a NoClassDefFoundError. Why? Well the Byteman exception classes are only visible to the system loader. The JVM will only resolve references from a given class via its loader. Since java.util.List was loaded via the bootstrap loader the JVM asks it to look up EarlyReturnException and, of course, it is not visible.
What happens if we boost the Byteman jar into the bootstrap classpath so it is is visible from the bootstrap loader but leave TracingHttpServerFilter and Grizzly in the system path. In that case when you inject a rule into java.util.List.append() the Byteman exception classes are resolved ok. Also, let's assume you inject a rule into a method of a system path class like FilterChainBuilder$StatelessFilterChainBuilder. That also resolves the Byteman exception classes because the system loader delegates the lookup to the bootstrap loader. So, goals 1 and 2 are not prejudiced. However, what about goals 3 and 4?
Goal 3 will not be met if you leave the jar containing TracingHttpServerFilter in the system path. The rule injected into java.util.List.append will fail. The target class for the rule is loaded via the bootstrap loader. So, when Byteman tries to resolve the reference to TracingHttpServerFilter via the bootstrap loader it will not be visible. n.b. it cannot simply pick some other loader to resolve classes mentioned in the rule because that might lead to type mismatches which could produce invalid bytecode.
However, goal 4 will still be met if you leave the jar containing TracingHttpServerFilter in the system path. The rule injected into StatelessFilterChainBuilder will resolve fine. StatelessFilterChainBuilder is loaded via the system loader and so the reference to TracingHttpServerFilter will find a class local to that loader.
Now, what happens if you try to fix goal 3 by promoting the jar containing TracingHttpServerFilter into the bootstrap path? Unfortunately, goals 3 and 4 will both fail for a different reason. The rule injected into java.util.List.append will have no problem finding TracingHttpServerFilter because it is now located in the bootstrap path. The rule injected into StatelessFilterChainBuilder will also be able to resolve the helper class since the lookup via the system loader will delegate to the bootstrap loader. However, TracingHttpServerFilter itself will fail to link properly when it is loaded. It references Grizzly classes that are only available in the system path. When the bootstrap loader tries to link TracingHttpServerFilteryou get a NoClassDefFoundError again.
Your last resort seems to be also to boost the Grizzly jar into the bootstrap path. That does allow TracingHttpServerFilter to resolve against the Grizzly classes, However, when you run the app you will probably find that some of the Grizzly jars reference classes that belong to other libraries that are also only visible via the system path. If you only exercise code in a few Grizzly classes you might get lucky and never try to load those libraries. However, you are on a slippery slope where you may end up having to boost the whole of Glassfish and your app into the bootstrap path. That way madness lies!!!
Ok, what else can you do? Well, if you look at the 4 goals here we satisfied 3 of them with the setup where we had Byteman in the bootstrap path and the jar containing TracingHttpServerFilter in the system path along with the Girzzly jars and the app jars. You can inject rules into system or bootstrap classes and you can use your Grizzly helper when injecting into system path classes, including Grizzly's own classes like StatelessFilterChainBuilder. Only goal 3 is missing.
So, let's see what we can do to fix this. First things first: is it broke? Do you actually need to invoke TracingHttpServerFilter and/or Grizzly functionality from rule injected into JDK code? If the answer is no then this setup should be ok. Just make sure that any rules which get injected into bootstrap classes don't attempt to refer to TracingHttpServerFilter or to Grizzly classes.
Alternatively, let's assume you do need to use functionality that belongs to class TracingHttpServerFilter from rules that target bootstrap classes. In that case you need to create a wrapper class that invokes the required functionality but does so without causing link errors. How do you do that? Load the class yourself and use reflection!
You will have to put the wrapper class into a jar that is inserted into the bootstrap path so your rules can see it. It will have to lookup class TracingHttpServerFilter explicitly via the system loader. Then it will have to find the methods it wants to call and invoke the required behaviour using reflection. So, you need something like this:
class TracingHttpServerFilterWrapper
{
static Class targetClass;
static Constructor targetConstructor
Method targetMethod;
. . .
static {
try {
targetClass = ClassLoader.findSystemClass("TracingHttpServerFilterWrapper");
} catch (Exception e) {
System.out.println("Oops, should not happen! " + e);
}
try {
for (Constructor constructor : targetClass.getDeclaredConstructors()) {
if (constructor.
}
} catch (Exception e) {
System.out.println("Oops, should not happen! " + e);
}
}
public Object createIndirectInstance(Type1 arg1, ... Typen argn)
{
try {
return targetClass.newInstance(arg1, ... argn);
} catch Exception e) {
System.out.println("Oops, should not happen! " + e);
return null;
}
}
public Object invokeTargetMethod(Object targetInstance, Type1 arg1, ... Typen argn)
{
try {
return targetMethod.invoke(targetInstance, arg1, ... argn);
} catch Exception e) {
System.out.println("Oops, should not happen! " + e);
return null;
}
}
}