1 2 Previous Next 16 Replies Latest reply on Sep 9, 2009 6:50 AM by Andrew Dinn

    instrumenting boot load classes such as java.io.File

    Stefan Egli Newbie

      I'm currently trying to get Byteman to instrument classes such as java.io.File, java.io.FileOutputStream etc. It looks like those classes are loaded *before* the Byteman's Main.premain is called - and according to the javadoc of Instrumentation.addTransformer you only get called in ClassFileTransformer.transform with classes that are loaded *after* that addTransformer call.

      I have had a look at what classes are loaded before addTransformer (using Instrumentation.getAllLoadedClasses() and the list is quite big. So I think it would be generally interesting to have a solution for this.

      At first I was mislead by my observation that Byteman's Main class imports java.io.File and that that would be the problem. But it looks like java.io.File is used way earlier in the JVM bootstrap process. Hence it wouldn't help removing that import.

      Any ideas for this issue?

      I tried going via Instrumentation.redefineClasses - but failed

      Cheers,
      Stefan

        • 1. Re: instrumenting boot load classes such as java.io.File
          Andrew Dinn Master

          Hi Stefan,

          Nice to hear that you are trying to use Byteman.

          There are indeed bootstrap issues when it comes to redefining classes which are needed by the JVM in order to get the Transformer loaded (also with modifying classes which are used to keep it running :-). Byteman is a powerful but dangerous tool.

          I am not certain whether such cases will be handled correctly in the latest build but I know it will definitely not work in Byteman 1.0.2. It was supposed to work in that release but I accidentally disabled injection of trigger code for classes loaded by the bootstrap loader.

          A fix for this was slipped into 1.0.3 -- I did not provide a corresponding JIRA so details of the fix do not appear in the release notes. With this fix in place I have been able to successfully transform the bootstrap-loaded class java.io.FileNotFoundException (this was under JDK 6). So, could I ask you to try again using the latest release (posted today) and see if you can get it to transform java.io.File? The class loader and transformer initialisation code may not actually be relying on this class (nor on java.io.FileOutputStream).

          If you find that rules are not injected into java.io.File etc by the current code then let me know or raise a BYTEMAN JIRA for it. I think I can fix this problem fairly easily using the retransform capability of the JDK 6 Instrumentation class. I currently have a prototype implementation based on JDK 6 which successfully retransforms previously compiled files as rules are loaded dynamically during program execution. It should be possible to scan the loaded classes list when the Transformer is created and simply force a retransform of any classes which match rules in the initial rule set. This is what I do anyway when rules are loaded dynamically by my prototype.

          This would mean that a rule which matches a pre-loaded class will not be triggered during initial bootstrap but it should ensure that it gets applied from the point where the transformer is installed. It also means that rules attached to the method will not be available (static initialisers don't get rerun after retransforming). Still, it ought to be possible to inject code for other cases.

          • 2. Re: instrumenting boot load classes such as java.io.File
            Andrew Dinn Master

            Sorry, some of my text got eaten. What I meant to say was

            "It also means that rules attached to the ***clinit*** method will not be available (static initialisers don't get rerun after retransforming). ..."

            • 3. Re: instrumenting boot load classes such as java.io.File
              Stefan Egli Newbie

              Hi Andrew,

              Thanks for that quick reply!

              I tried 1.0.3 - but it wouldn't instrument that java.io.File (exists() method) I'm afraid. This is my rule:

              RULE fileexists
              CLASS java.io.File
              METHOD exists
              AT ENTRY
              BIND NOTHING
              IF true
              DO debug("File was called with exists")
              ENDRULE

              Should be fairly straight-forward - it works fine when I use non JDK classes.

              Would be cool if you could get that work with the redefineClasses variant.

              Cheers,
              Stefan

              • 4. Re: instrumenting boot load classes such as java.io.File
                Andrew Dinn Master

                Ok, I'll modify my prototype to redefine loaded classes at transformer install time and see if it will fix this.

                I'll be checking my prototype code into trunk in the next few days anyway. So, if it works you may be able to do what you want (so long as you are happy playing with the trunk build).

                • 5. Re: instrumenting boot load classes such as java.io.File
                  Stefan Egli Newbie

                  Yea sure, I'm fine with the trunk. Just ping me if you don't mind when you've got something for me to test.

                  PS: Will you be fixing this then eventually into a 1.0.4?

                  • 6. Re: instrumenting boot load classes such as java.io.File
                    Andrew Dinn Master

                     


                    Yea sure, I'm fine with the trunk. Just ping me if you don't mind when you've got something for me to test.


                    Ok, I'll reply to this thread.


                    PS: Will you be fixing this then eventually into a 1.0.4?


                    No, the retransformer version will be the first of the 1.1 releases (1.1.0)


                    • 7. Re: instrumenting boot load classes such as java.io.File
                      Stefan Egli Newbie

                      just a quick FYI: I tried using redefineClasses but failed - maybe I was using the wrong set of byte[] for the class - or the wrong ClassLoader. As those early loaded classes get loaded by the bootstrap classloader which is not returned when you do a class.getClassLoader() (==null) - maybe Byteman can't deal with that ?

                      • 8. Re: instrumenting boot load classes such as java.io.File
                        Andrew Dinn Master

                         


                        just a quick FYI: I tried using redefineClasses but failed - maybe I was using the wrong set of byte[] for the class - or the wrong ClassLoader. As those early loaded classes get loaded by the bootstrap classloader which is not returned when you do a class.getClassLoader() (==null) - maybe Byteman can't deal with that ?


                        The problem with using the redefine capability is that you have to have the original class bytes available. SInce the Transformer will not have seen java.io.File this is not an option. That's why I chose to use the retransform capability. You merely have to pass the class(es) you want redefined to the Instrumentation instance and the system identifies the relevant bytecodes and passes them back to your transformer. You can see how this works when I upload my prototype.

                        The null ClassLoader issue is something I had to resolve in the 1.0.3 version. I use the system ClassLoader to access related classes if I am trying to inject code into a bootstrap class. Strictly this may risk invalid class exceptions because the system ClassLoader can be redefined when the JVM is started. This probably also constitutes a security hole (so, just add it to the list :-). However, for the default implementation of the system ClassLoader or for a well-behaved substitute this will provide correct access to all the required bootstrap classes.

                        Another wrinkle is that you also need the agent premain routine to append the byteman jar to the system classpath (another JDK6 only option) in order for redefine/retransform to work. Otherwise the injected code fails with an unresolved reference to the Byteman exception classes, ExecuteException, ReturnException and ThrowException.

                        By the way there is still an annoying issue with using redefine in the Sun JDK/OpenJDK. These JVMs do not normally cache bytecode for loaded classes. They do if a transformer has previously successfully modified the class but they throw away the class bytes if the class was not transformed at first load. In the latter case when a retransform request is made the JVM reconstitutes the bytecode from the VMs in-memory klass object. This would be ok except that the current JDK 6/7 reconstitute routine does not regenerate local variable tables. So, my dynamic rule upload currently does not work for rules which use local variables. I have patched and avlidated this in my own OpenJDK 7 tree and raised a bugzilla (#100100) for it wioth the patch attached but this has not yet been incorporated into the JVM and made available for general release.


                        • 9. Re: instrumenting boot load classes such as java.io.File
                          Andrew Dinn Master

                          The following JIRA has been raised, fixed and closed:

                          https://jira.jboss.org/jira/browse/BYTEMAN-25

                          The fixes for this and the underlying redefine capability on which it depends are now available in trunk. Details of how to enable and use the redefine capability are in the latest manual in trunk.

                          Enjoy :-)

                          • 10. Re: instrumenting boot load classes such as java.io.File
                            Stefan Egli Newbie

                            Great! I'll have to wait until Monday for testing it though - will let you know. Cheers, Stefan

                            • 11. Re: instrumenting boot load classes such as java.io.File
                              Stefan Egli Newbie

                              Hi Andrew,

                              Great! Got it to work with the trunk!

                              Two things which I noticed though:

                              * The JVM param you added to JIRA 25: I had to set it to something like this instead: -javaagent:byteman.jar=redefine:true,boot:byteman.jar,script:testbytemanrules.txt
                              (Not sure about whether boot:byteman.jar is really necessary though :S )

                              * I first stumbled over the fact that the following rule wouldn't work:

                              RULE fileexists
                              CLASS java.io.File
                              METHOD exists
                              AT ENTRY
                              BIND NOTHING
                              IF true
                              DO debug("File was called with exists")
                              ENDRULE

                              When I changed the 'AT ENTRY' to 'AT EXIT' it worked! Looks like visitEnd() in LineCheckMethodAdapter doesn't call into setVisitOk() which then yields no trigger points. Could that because of using classes without line numbers?

                              Cheers,
                              Stefan

                              • 12. Re: instrumenting boot load classes such as java.io.File
                                Andrew Dinn Master

                                 


                                Great! Got it to work with the trunk!


                                Excellent.


                                * The JVM param you added to JIRA 25: I had to set it to something like this instead: -javaagent:byteman.jar=redefine:true,boot:byteman.jar,script:testbytemanrules.txt
                                (Not sure about whether boot:byteman.jar is really necessary though :S )


                                Hmm, that's sort of what I intended with my comment in the JIRA. You were meant to substitute the bits which looked like <...> as you have done here.

                                The boot:byteman.jar option is only needed if you are injecting rule code into JVM bootstrap loader classes (like java.io.File) and byteman.jar. This is only necessary if byteman.jar is not already in the JVM boot classpath. I guess if you add . to CLASSPATH and put byteman.jar is in the current directory then it will not be necessary.


                                * I first stumbled over the fact that the following rule wouldn't work:

                                RULE fileexists
                                CLASS java.io.File
                                METHOD exists
                                AT ENTRY
                                BIND NOTHING
                                IF true
                                DO debug("File was called with exists")
                                ENDRULE

                                When I changed the 'AT ENTRY' to 'AT EXIT' it worked! Looks like visitEnd() in LineCheckMethodAdapter doesn't call into setVisitOk() which then yields no trigger points. Could that because of using classes without line numbers?


                                Yes, that looks like a problem with not having line number info in your compiled code. It happens because I did something a little cheap and cheeky to implement AT ENTRY. An AT ENTRY location is implemented by treating it as if it were AT LINE -1 and reusing the LineCheck/TriggerAdapter and LineCheck/TriggerMethodAdapter implementation. This works because it causes the trigger call to be injected at the first line of the method (whatever line it starts on the first line numver > -1). However, if there is no line number info then LineCheckMethodAdapter.visitLineNumber ()never gets called which means setTriggerPoint() never gets called and hence (in visitEnd) checkBindings() fails and setVisitOk() is not called

                                What I really ought to do is implement a proper EntryCheckAdapter and EntryTriggerAdapter which do the job properly. I will raise a JIRA for this and fix it. For now you can work round this by compiling your java program with javac -g (or javac -g:lines).

                                • 14. Re: instrumenting boot load classes such as java.io.File
                                  Stefan Egli Newbie

                                  Awesome! Just checked your fix for BYTEMAN-27 - works smooth as!

                                  I have only one input for the docu (it might just be me though not having figured it out myself): It was not rightly intuitive how you'd have to define constructor signatures if you had more than one - i.e. using
                                  METHOD (String,boolean).
                                  Maybe it would be worth a sidenote in the docu.

                                  Otherwise, got everything working with byteman as intended! Probably gonna use it extensively as quality assurance and paranoia checking tool!

                                  Cheers,
                                  Stefan

                                  1 2 Previous Next