Instrumenting classes from both boot and non-boot in Tomcat

[mthurmaier]

Hi,  I'm having a problem configuring the -javaagent flag in the JAVA_OPTS to Tomcat for loading BYTEMAN agent so that it can see rules for classes loaded at both boot and non-boot.

Here are a couple rules (re-posed from a previous discussion):

RULE trace sendFileToResponse exit

CLASS ConfigurationManagerServlet

METHOD sendFileToResponse

AT ENTRY

IF true

DO traceln(">>>>>>>>>>>>>>>>>>>entering sendFileToResponse")

ENDRULE

RULE trace sendFileToResponse exit

CLASS ConfigurationManagerServlet

METHOD sendFileToResponse

AT EXIT

IF true

DO traceln(">>>>>>>>>>>>>>>>>>>exiting sendFileToResponse")

ENDRULE

RULE cause sendFileToResponse exception1

CLASS File

METHOD <init>(String)

AT ENTRY

IF callerEquals("ConfigurationManagerServlet.sendFileToResponse", true)

DO traceln(">>>>>>>>>>>>>>>>>>File throwing IOException");

   throw new java.lang.NullPointerException("byteman forcing File exception")

ENDRULE

I have tried the following, with corresponding results:

• JAVA_OPTS="-Dorg.jboss.byteman.verbose -Dorg.jboss.byteman.transform.all -javaagent:/usr/share/apache-tomcat/byteman/lib/byteman.jar=boot:/usr/share/apache-tomcat/byteman/lib/byteman.jar,listener:true,port:9091,address:localhost  -Xss256K -Xmx2048m -XX:MaxMetaspaceSize=512m" 
  • --- RESULT:
    • I see all rules loaded with bmsubmit.sh
    • I see log entries in "catalina.server.out" showing the rules being inserted (e.g.

TransformListener() : handling connection on port 9091

retransforming com.vmware.vlvma.vla.rest.ConfigurationManagerServlet

retransforming java.io.File

org.jboss.byteman.agent.Transformer : possible trigger for rule trace sendFileToResponse entry in class com.vmware.vlvma.vla.rest.ConfigurationManagerServlet

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into com.vmware.vlvma.vla.rest.ConfigurationManagerServlet.sendFileToResponse(java.lang.String,javax.servlet.http.HttpServletResponse) void for rule trace sendFileToResponse entry

org.jboss.byteman.agent.Transformer : inserted trigger for trace sendFileToResponse entry in class com.vmware.vlvma.vla.rest.ConfigurationManagerServlet

org.jboss.byteman.agent.Transformer : possible trigger for rule trace sendFileToResponse exit in class com.vmware.vlvma.vla.rest.ConfigurationManagerServlet

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into com.vmware.vlvma.vla.rest.ConfigurationManagerServlet.sendFileToResponse(java.lang.String,javax.servlet.http.HttpServletResponse) void for rule trace sendFileToResponse exit

org.jboss.byteman.agent.Transformer : inserted trigger for trace sendFileToResponse exit in class com.vmware.vlvma.vla.rest.ConfigurationManagerServlet

org.jboss.byteman.agent.Transformer : possible trigger for rule cause sendFileToResponse exception2 in class com.vmware.vlvma.vla.rest.ConfigurationManagerServlet

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into com.vmware.vlvma.vla.rest.ConfigurationManagerServlet.sendFileToResponse(java.lang.String,javax.servlet.http.HttpServletResponse) void for rule cause sendFileToResponse exception2

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into com.vmware.vlvma.vla.rest.ConfigurationManagerServlet.sendFileToResponse(java.lang.String,javax.servlet.http.HttpServletResponse) void for rule cause sendFileToResponse exception2

org.jboss.byteman.agent.Transformer : inserted trigger for cause sendFileToResponse exception2 in class com.vmware.vlvma.vla.rest.ConfigurationManagerServlet

org.jboss.byteman.agent.Transformer : possible trigger for rule cause sendFileToResponse exception1 in class java.io.File

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into java.io.File.<init>(java.lang.String) void for rule cause sendFileToResponse exception1

org.jboss.byteman.agent.Transformer : inserted trigger for cause sendFileToResponse exception1 in class java.io.File

Rule.execute called for cause sendFileToResponse exception1_3

HelperManager.install for helper class org.jboss.byteman.rule.helper.Helper

calling activated() for helper class org.jboss.byteman.rule.helper.Helper

Default helper activated

• byteman never prints the "entering" or "exiting" messages.
• byteman regularly tests the condition for throwing the exception, but never prints the message indicating that it is throwing the exception, or throws the exception.
• I do not see any tracebacks in the log files for byteman
• I THINK that somehow byteman is not "seeing" the non-boot symbols at runtime. Just a guess. Andrew Dinn or someone may have a better idea.

• JAVA_OPTS="-Dorg.jboss.byteman.verbose -Dorg.jboss.byteman.transform.all -javaagent:/usr/share/apache-tomcat/byteman/lib/byteman.jar=listener:true,port:9091,address:localhost  -Xss256K -Xmx2048m -XX:MaxMetaspaceSize=512m"  NOTE: this is the SAME string, except that I'm not using '=boot:/path'.  NOTE2: I removed the rule for File to prevent byteman from throwing an exception (see below)
  • Results
    • Same resulst form bmsubmit.sh commands and entries in logs
    • When the code hits the method, byteman DOES print the "entering" and "exiting" messages. SO, byteman is NOW seeing the non-boot symbols, taking appropriate action.
    • IF I include the "exception" rule, byteman will throw the class-no-found error we would expect since File is loaded at boot and, without the "boot" option, can't see File.
    • Key here, these rules work when there is no "=boot" option.

So, the question is, What is the RIGHT JAVA_OPTS string to be able to get these rules to work for both boot and non-boot classes?

Final note: i'm currently using 3.0.10.

Thank you VERY much in advance.

Cheers!

[ochaloup]

Hi Matthew,

I'm not able to bring here any explicit explanation (that will need to wait for Andrew), just I would like point down few notes coming to my mind

* I have created a simple test-case where I use bmunit and changing a java lang class (boot and transform all options are needed to get it instrumented),plus having a rule that changes how the application code works. It's a simple chunk of code (a main method) but the transformation is done on "boot" and "non-boot" classes and works fine.

* You can see your example ruleset contains a rule named 'trace sendFileToResponse exit' twice (twice named with 'exit', never named with 'entry') - that's just a typo, ritght?

* there could be seen the rule 'sendFileToResponse exception2' in the log - is not mentioned in the provided ruleset, does it do something what would be good to see?

* when you say  that rule 'Rule.execute called for cause sendFileToResponse exception1_3' is tries to be executed and there is no traceln message shown in the log - - could happen that there is some different code branch being executed in such case?
I do understand, you wrote that you can see different behavior in case the Byteman 'boot' option is used contra when it's not used but... If the 'RULE cause sendFileToResponse exception1' is executed but the 'IF' cause is evaluated to 'false' (probably as NullPointerException is not thrown) then it could be useful to know what is the stacktrace of the callee - what about to create a new rule with 'IF true" and action as 'DO traceStack()' to check what was executed?

Ondra

[adinn]

Hi Matthew,

Thanks for reporting this issue. I'm unclear as to what might be causing the behaviour you are seeing. I am also slightly confused as regards your description of the way you have set up your test/program run and your explanation of what is happening when it runs.

Firstly, you mention "rules for classes loaded at both boot and non-boot". Do you mean that you are loading Byteman scripts on the Java command line and also loading rules using BMUnit? (r perhaps bmsubmit?) Or do you mean that the rules provided on the Java command line apply to classes which are in the boot image and also to classes which are not in the boot image but rather loaded from the classpath?

Second, you say "I see all rules loaded with bmsubmit.sh". Are you really running the bmsubmit script? If so when? And how does that relate to what you do to exercise the relevant code running inside Tomcat.

Thirdly, you say "byteman regularly tests the condition for throwing the exception, but never prints the message indicating that it is throwing the exception, or throws the exception". How do you know Byteman is doing that? Have you observed the call in a debugger?

Perhaps you could provide a more precise description of what commands you run to start your application and exercise it to test the behaviour your rules are meant to observe. If you could also describe any use you make of bmsubmit and how that relates to the timing of the calls the rules are meant to observe that would be very helpful.

Another thing you might want to try is attaching a debugger to Tomcat and making sure that method ConfigurationManagerServlet.sendFileToResponse really is getting called in the case where you have employed the boot: option. You could also add a break to Byteman method Rule.execute to see if it is being called.

Finally, it would be helpful if you could get Byteman to dump any transformed bytecode so I can see whether rule injection is being performed as expected. The messages in the log file suggest that class ConfigurationManagerServlet is being transformed. If it is ok for me to see the bytecode for this class then I can verify that Byteman is injecting the expected code into it. If so then please add the following argument to your java command line

-Dorg.jboss.byteman.dump.generated.classes

This should cause Byteman to dump bytecode for the transformed class in the working directory your app is running in. The dump generates the required directory hierarchy appropriate to the transformed class's package. So, you should see the file written as <workdir>/org/my/package/tree/ConfigurationManagerServlet.class. Run command

javap -c -v -classpath <workdir> org.my.package.tree.ConfigurationManagerServlet

paste the output into a file (or at least the output for method sendFileToResponse) and attach it so I can check the transformation has been applied correctly.

Meanwhile, I'll check to see if I have any problems when passing the boot: option to  -javaagent running Byteman 3.0.10 in something a bit more simple than Tomcat. That might help pin down where the problem lies.

regards,

Andrew Dinn

[mthurmaier]

Hi Ondra and Andrew, I'll atteempt to answer both of your questions in one reply.

Ondra: point 1 and Andrew Point 1 (firstly): I'm not using "bmunit". I have a bunch of Java code running in Tomcat. So, I'm using "java options" ("rules" was a misnomer, sorry) to load BYTEMAN as part of the Tomcat start, with the listener option, so that I can load byteman-rules dynamically with the "bmsubmit.sh".  I hope that clarifies the situation for both of your first points.

Ondra: Point 2 (2 exits, no entry?): Yes, unfortunate copy/paste error. I do have 2 separate rules, one for entry, one for exit:

RULE trace sendFileToResponse entry

CLASS ConfigurationManagerServlet

METHOD sendFileToResponse

AT ENTRY

IF true

DO traceln(">>>>>>>>>>>>>>>>>>>>entering sendFileToResponse")

ENDRULE

RULE trace sendFileToResponse exit

CLASS ConfigurationManagerServlet

METHOD sendFileToResponse

AT EXIT

IF true

DO traceln(">>>>>>>>>>>>>>>>>>>exiting sendFileToResponse")

ENDRULE

Ondra: Point 3 (what does exception2 look like): The exception2 rule looks like this:

RULE cause sendFileToResponse exception2

CLASS ConfigurationManagerServlet

METHOD sendFileToResponse

AT WRITE $idx1 ALL

IF true

DO traceStack(">>>>>>>>>>>>>>>>> writing idx1");

ENDRULE

Ondra point 3 (can it be executed elsewhere?): I'm sure it is being executed in LOTS of places, because I'm instrumenting the File() object constructor. I only want to throw the exception, though, if the constructor is called from my particular function "ConifugrationManagerServlet.sendFileToResponse", which is why I have the IF clause that checks for that.

Andrew Point 2 (are you using bmsubmit): Yes, I have the following JAVA_OPTS in the Tomcat start: ".... -javaagent:......,listener", and then I use "bmsubmit.sh -l rules.bmt" The output of the command shows that it read the rules. If I just run "bmsubmit.sh" (since we're using the default port) it shows the rules that are loaded.   How does that relate?  I'm doing these steps:

1) Start Tomcat (with BYTEMAN loaded via JAVA_OPTS

2) Check that java agent is listening via "lsof -i:9091"

3) Submit my rule file via "${BYTEMAN_HOME}/bin/bmsubmit.sh -l rules.bmt"

4) Check that the rules are loaded with same command without "-l rules.bmt"

5) Start doing tail -f on appropriate log files

6) Drive the web-app over the appropriate code and watch the logs.

Andrew Point 3 (how do you know its hitting the rule?): I thought that the log message:

Rule.execute called for cause sendFileToResponse exception1_3

HelperManager.install for helper class org.jboss.byteman.rule.helper.Helper

calling activated() for helper class org.jboss.byteman.rule.helper.Helper

Default helper activated

Indicated that the rule was triggered. There are LOTS of those messages in my log file as I drive the code up to and over the instrumented code. It makes sense as just getting to the function that I'm trying to test causes lots of files to get opened, all of which will call the File() object's constructor, which is what the exception1 rule is about.

Andrew Point 4: I'll check with higher-pay-grade folks to see if I can get you the transformed bytecode.

Thanks for all the help!

[mthurmaier]

OK. Maybe I'm an idiot. I just realized that I had not set BYTEMAN_HOME variable in the environment of the Tomcat process. I did that, and now I seem to get full functionality. I'm getting the entry and exit messages and the exception thrown when I hit the File() constructor from the function I'm instrumenting, etc.  What worked for me was the following environment variable settings:

-------------

BYTEMAN_HOME=/usr/share/apache-tomcat/byteman

JAVA_OPTS="-Dorg.jboss.byteman.verbose -Dorg.jboss.byteman.transform.all -javaagent:/usr/share/apache-tomcat/byteman/lib/byteman.jar=boot:/usr/share/apache-tomcat/byteman/lib/byteman.jar,listener:true,port:9091,address:localhost  -Xss256K -Xmx2048m -XX:MaxMetaspaceSize=512m"

------------

NOTE: these variables are in a file that is read by the Tomcat server start script. That code exports all these variables before launching Tomcat.

Perhaps you can enhance the documentation with this example for launching byteman from Tomcat (or other JVM-based application servers)?

Thank you VERY much for your other inputs. I think I'm now up and running! 

[adinn]

Hi Matthew,

I'm glad to hear you are now running ok. I'm not actually sure why it is now working though :-/

Environment variable BYTEMAN_HOME is never dereferenced by the agent code so I don't really understand why setting it in the Tomcat environment has any effect.

It is dereferenced by the bin scripts bmjava., bmcheck, bmsubmit and bminstall to locate the locally installed byteman jars (although in the case of bmsubmit it is only to locate bmsubmit.jar which doesn't actually use any of the code provided in byteman.jar).

It is also dereferenced by class Install to identify the location of the agent jar byteman.jar -- although if BYTEMAN_HOME is unset then Install will search the classpath for a jar with name byteman.jar or, alternatively, byteman-x.y.z.jar.

So, whether you set BYTEMAN_HOME or leave it unset really shouldn't change how the Byteman agent runs inside Tomcat. <scratches_head/>

I'm happy to leave question as resolved since clearly you now have a setup that is working. However, I suspect something else may be involved in causing the original problem you saw. So, just in case: i) if you have any further problems then let me know right away ii) be sure to keep track of anything you might have changed in your setup should a problem arise. I'm very keen to dig deeper into this if more evidence turns up (it's always good to know for sure what is going on in case a bigger problem is being masked).

Also, if you have any other issues using Byteman or need help writing some more complex rule sets don't hesitate to ask here. Ondra and I are always interested to hear what users are up to and help them apply Byteman to new problems. As also are some of the other users who hang out here.

Thanks for reporting this issue and I hope Byteman answers your needs.

[mthurmaier]

Given you explanation, I don't know why it is working now, either. But I'm ecstatic that it is.

Next, I learn BMunit and TestNG frameworks and how to put those together with byteman. We have some code running in Tomcat, some from the CLI. We need a good java unit test tool to cover non-golden-path cases to increase code coverage. BYTEMAN now looks promising.

Thank you both very much for your swift attention to my questions. I really appreciate it.