3 Replies Latest reply on Dec 5, 2016 6:27 AM by Andrew Dinn

    Trying to instrument the Command Line with Byteman

    Francesco Marchioni Apprentice

      Hi all,

      I'm trying to add some trace functionalities to the application server's CLI with Byteman (3.0.3). I'm having some troubles in getting my rules executed on the entry point (which is org.jboss.as.cli.CommandLineMain).

      Here is my minimalist rule, on which I'll need to build up:

      RULE trace main entry
      
      CLASS org.jboss.as.cli.CommandLineMain
      
      METHOD main
      
      AT ENTRY
      IF true
      
      DO traceln("Byteman detected you are entering main")
      
      ENDRULE
      

       

      Launching with Debug on, I have the following output:

       

      $ jboss-cli.sh 
      
      org.jboss.byteman.agent.Transformer : possible trigger for rule trace main entry in class org.jboss.as.cli.CommandLineMain
      RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.jboss.as.cli.CommandLineMain.main(java.lang.String[]) void for rule trace main entry
      org.jboss.byteman.agent.Transformer : inserted trigger for trace main entry in class org.jboss.as.cli.CommandLineMain
      Rule.execute called for trace main entry_0
      Rule.execute for decommissioned key trace main entry_0
      
      You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
      [disconnected /] 
      

      However, even if the main method has been executed (from the stacktrace on visualvm), there is no output produced by the rule.

      Here's the JAVA_OPTS I'm passing to jboss-cli.sh:

      JAVA_OPTS="-Dorg.jboss.byteman.verbose -Dorg.jboss.byteman.transform.all -javaagent:${BYTEMAN_HOME}/lib/byteman.jar=script:monitor.btm"
      

      Any idea what's wrong with it ?

        • 1. Re: Trying to instrument the Command Line with Byteman
          Andrew Dinn Master

          Hi Francesco,

           

          This bit of the trace suggests what may be going wrong:

           

          Rule.execute called for trace main entry_0  
          Rule.execute for decommissioned key trace main entry_0  
          

           

          The first message implies that the rule has been injected into the target method and the injected code which initiates execution of the rule code has been reached and ruel execution has been triggered.

           

          The second message implies that the injected trigger code has tried to look up  the rule it is associated with and not found it.

           

          Now that can happen for a legitimate reason but only when when a rule is first installed and then uninstalled. The uninstall operation removes the rule from list of active rules and then retransforms the trigger method to remove the injected  code. In this situation it sometimes happens that the transformed version of the method code is still being executed by  some thread. So, when the trigger code is reached the lookup fails. That wil not be what is happening here though. Firstly, it's a fairly rare circumstance dependent on very specific timing. Secondly, your rule has not been uninstalled so this case will not apply.

           

          There is also another illegitimate circumstance where this can happen and I think that is what is going on here. The trigger code that gets injected by Byteman employs a unique String key to look up the rule. When the method is transformed the rule is inserted into a hash table, a static field of Byteman class Transformer, under that key. Class Transformer will have been defined in either the system loader or the bootstrap loader depending upon how you deployed Byteman. Now when the trigger code gets executed it uses a static method of class Rule to look up the rule, passing the String key as an argument. Class Rule gets resolved from the classloader of the trigger method. If your deployment is configured incorrectly and there is another copy of the Byteman jar in the system then you can end up seeing a different version of class Transformer loaded by a different classloader· This version will have a static hash table field which is empty.

           

          One common thing which can cause this to happen is when the application code itself includes references to Byteman classes but that normally only causes a problem wen you install the Byteman jar into the bootstrap path using the boot: javaagent option. I cannot really be sure what is going wrong here unless you can describe how the classpath and module configuration is defined for your deployment.

           

          Does the CLI code employ your JBoss Modules module hierarchy? If so then you probably need to redirect all attempts to load the byteman jar to the sys loader using -Djboss.modules.system.pkgs="org.jboss.byteman.*". However, that may not be what is going wrong here.

           

          regards,

           

           

          Andrew Dinn

           

          • 2. Re: Trying to instrument the Command Line with Byteman
            Francesco Marchioni Apprentice

            Thanks Andrew, you put me on the right track. I didn't notice that the jboss-cli script includes this line where the jboss.modules.system.pkgs redefines what I have set earlier. (The purpose of it is to set some UI modules for the graphical mode of the CLI)

            if $darwin ; then
                # Add the apple gui packages for the gui client
                JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=com.apple.laf,com.apple.laf.resources"
            else
                # Add base package for L&F
                JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=com.sun.java.swing"
            fi
            

            This causes an issue with the classloader (EarlyReturnException) which I (wrongly) managed to solve by adding byteman.jar to the org.jboss.cli module.xml file. The side effect of this is that the Rule was unloaded as you could see from my post.

            Now it works, thanks for the hint!

            Francesco

            • 3. Re: Trying to instrument the Command Line with Byteman
              Andrew Dinn Master

              Hi Francesco,

               

              Thanks for identifying this problem. I'm glad that Byteman is now working.

               

              Ideally the jboss-cli script ought to test whether jboss.modules.system.pkgs is already set and if so append the extra packages to the existing ones. I'll raise an issue for that with the relevant JBoss devs.

               

              regards,

               

              Andrew Dinn