3 Replies Latest reply on Dec 5, 2016 6:27 AM by adinn

Trying to instrument the Command Line with Byteman

f_marchioni Dec 3, 2016 8:54 AM

Hi all,

I'm trying to add some trace functionalities to the application server's CLI with Byteman (3.0.3). I'm having some troubles in getting my rules executed on the entry point (which is org.jboss.as.cli.CommandLineMain).

Here is my minimalist rule, on which I'll need to build up:

RULE trace main entry

CLASS org.jboss.as.cli.CommandLineMain

METHOD main

AT ENTRY
IF true

DO traceln("Byteman detected you are entering main")

ENDRULE

Launching with Debug on, I have the following output:

$ jboss-cli.sh 

org.jboss.byteman.agent.Transformer : possible trigger for rule trace main entry in class org.jboss.as.cli.CommandLineMain
RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.jboss.as.cli.CommandLineMain.main(java.lang.String[]) void for rule trace main entry
org.jboss.byteman.agent.Transformer : inserted trigger for trace main entry in class org.jboss.as.cli.CommandLineMain
Rule.execute called for trace main entry_0
Rule.execute for decommissioned key trace main entry_0

You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /]

However, even if the main method has been executed (from the stacktrace on visualvm), there is no output produced by the rule.

Here's the JAVA_OPTS I'm passing to jboss-cli.sh:

JAVA_OPTS="-Dorg.jboss.byteman.verbose -Dorg.jboss.byteman.transform.all -javaagent:${BYTEMAN_HOME}/lib/byteman.jar=script:monitor.btm"

Any idea what's wrong with it ?

1. Re: Trying to instrument the Command Line with Byteman

adinn Dec 4, 2016 5:51 AM (in response to f_marchioni)
Hi Francesco,

This bit of the trace suggests what may be going wrong:

Rule.execute called for trace main entry_0 Rule.execute for decommissioned key trace main entry_0

The first message implies that the rule has been injected into the target method and the injected code which initiates execution of the rule code has been reached and ruel execution has been triggered.

The second message implies that the injected trigger code has tried to look up the rule it is associated with and not found it.

Now that can happen for a legitimate reason but only when when a rule is first installed and then uninstalled. The uninstall operation removes the rule from list of active rules and then retransforms the trigger method to remove the injected code. In this situation it sometimes happens that the transformed version of the method code is still being executed by some thread. So, when the trigger code is reached the lookup fails. That wil not be what is happening here though. Firstly, it's a fairly rare circumstance dependent on very specific timing. Secondly, your rule has not been uninstalled so this case will not apply.

There is also another illegitimate circumstance where this can happen and I think that is what is going on here. The trigger code that gets injected by Byteman employs a unique String key to look up the rule. When the method is transformed the rule is inserted into a hash table, a static field of Byteman class Transformer, under that key. Class Transformer will have been defined in either the system loader or the bootstrap loader depending upon how you deployed Byteman. Now when the trigger code gets executed it uses a static method of class Rule to look up the rule, passing the String key as an argument. Class Rule gets resolved from the classloader of the trigger method. If your deployment is configured incorrectly and there is another copy of the Byteman jar in the system then you can end up seeing a different version of class Transformer loaded by a different classloader· This version will have a static hash table field which is empty.

One common thing which can cause this to happen is when the application code itself includes references to Byteman classes but that normally only causes a problem wen you install the Byteman jar into the bootstrap path using the boot: javaagent option. I cannot really be sure what is going wrong here unless you can describe how the classpath and module configuration is defined for your deployment.

Does the CLI code employ your JBoss Modules module hierarchy? If so then you probably need to redirect all attempts to load the byteman jar to the sys loader using -Djboss.modules.system.pkgs="org.jboss.byteman.*". However, that may not be what is going wrong here.

regards,

Andrew Dinn
Actions
2. Re: Trying to instrument the Command Line with Byteman

f_marchioni Dec 5, 2016 3:38 AM (in response to adinn)
Thanks Andrew, you put me on the right track. I didn't notice that the jboss-cli script includes this line where the jboss.modules.system.pkgs redefines what I have set earlier. (The purpose of it is to set some UI modules for the graphical mode of the CLI)
if $darwin ; then # Add the apple gui packages for the gui client JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=com.apple.laf,com.apple.laf.resources" else # Add base package for L&F JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=com.sun.java.swing" fi
This causes an issue with the classloader (EarlyReturnException) which I (wrongly) managed to solve by adding byteman.jar to the org.jboss.cli module.xml file. The side effect of this is that the Rule was unloaded as you could see from my post.
Now it works, thanks for the hint!
Francesco
Actions
3. Re: Trying to instrument the Command Line with Byteman

adinn Dec 5, 2016 6:27 AM (in response to f_marchioni)

Hi Francesco,

Thanks for identifying this problem. I'm glad that Byteman is now working.

Ideally the jboss-cli script ought to test whether jboss.modules.system.pkgs is already set and if so append the extra packages to the existing ones. I'll raise an issue for that with the relevant JBoss devs.

regards,

Andrew Dinn
Actions

Go to original post