3 Replies Latest reply on Apr 27, 2016 4:17 AM by adinn

    Errors in simple fault injection with TestNG and maven

    andou528

      Hi All :

       

      I am trying to modify the TestNG test case in the tutorial "BMUnit : Using Byteman with JUnit or TestNG from maven and ant".

       

      But there some errors I cannot figure it out.

       

      The modified method openFile():

      public PrintStream openFile() {
              File file = new File(filename);
              try {
                  FileOutputStream fos = new FileOutputStream(file);
                  PrintStream ps = new PrintStream(fos, true);
                  testVariable = false;
                  return ps;
              } catch (FileNotFoundException e) {
                  System.out.println("Unable to open file " + file.getName());
                  System.out.println(e);
                  testVariable = true;
                  return null;
              }
          }
      

       

      The modified test case:

      @Test
      @BMRule(name = "test exception catch",
                  targetClass = "org.my.app.WebWriter",
                  targetMethod = "openFile()",
                  targetLocation = "AFTER WRITE $testVariable",
                  action = "throw new java.io.FileNotFoundException( \"Ha ha Byteman fooled you again!\" )"
                  )
          public void testExceptionCatch()
          {
              System.out.println("-------- testExceptionCatch start ---------");
              WebWriter writer = new WebWriter("foo.html", "Andrew");
              PrintStream ps = writer.openFile();
              Assert.assertTrue(ps == null);
              System.out.println("-------- testExceptionCatch end ---------\n");
          }
      

       

      What I want is changing the place to inject the fault.

      But I get the following errors:

      
      -------------------------------------------------------
       T E S T S
      -------------------------------------------------------
      Running TestSuite
      byteman jar is /Users/AAA/.m2/repository/org/jboss/byteman/byteman/2.2.1/byteman-2.2.1.jar
      objc[92708]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/jre/bin/java and /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined.
      Setting org.jboss.byteman.allow.config.update=true
      Setting org.jboss.byteman.debug=true
      Setting org.jboss.byteman.verbose=true
      TransformListener() : accepting requests on localhost:9091
      TransformListener() : handling connection on port 9091
      -------- testExceptionCatch start ---------
      org.jboss.byteman.agent.Transformer : possible trigger for rule test exception catch in class org.my.app.WebWriter
      org.jboss.byteman.agent.Transformer : inserted trigger for test exception catch in class org.my.app.WebWriter
      TransformListener() : handling connection on port 9091
      retransforming org.my.app.WebWriter
      TransformListener() : handling connection on port 9091
      Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.95 sec <<< FAILURE!
      
      Results :
      
      Failed tests: 
        testExceptionCatch(org.my.WebWriterTest3)
      
      Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
      

       

      I found this post "Can't inject RuntimeException: ensureTypeCheckedCompiled fails" might be useful. But it says the bug is closed.

      https://issues.jboss.org/browse/BYTEMAN-173

       

      Can anyone give me some hints? Thank you.

        • 1. Re: Errors in simple fault injection with TestNG and maven
          adinn

          Hi Andrew,

           

          First of all, thanks for reporting this problem. It's always good to hear from users and find out about any issues they are facing (although I also like to hear about successes :-).

           

          I am not completely clear what your test program is doing. There is an assignment on line 6 of your first listing

           

          testVariable = false;
          


          I don't see any declaration for testVariable. Is it a field of this?


          Your RULE appears to be trying to inject at the point where a local variable is assigned:


           

          targetLocation = "AFTER WRITE $testVariable",


          The $varname syntax is used to reference parameters and local variables that are in scope (as well as for special vars). If you really meant testVariable to be a field then the location should be:


          targetLocation = "AFTER WRITE testVariable",


          Indeed, if you want to make it explicit that you are interested in writes to field testVariable of class WebWriter, you would write:


           

          targetLocation = "AFTER WRITE WebWriter.testVariable",

           

          That said, I'm unclear whether this is the cause of your problem because I can see that your rule is being injected. I would not have expected injection to proceed when testVariable is a  field and your rule tries to refer to a local var with that name. So, I'll need you to clarify exactly what is happening here.

           

          Also, could you append the test output file for the failing test. That should include trace showing the rule being type-checked and executed (assuming it gets to either of those stages) which should help enormously to diagnose the problem.

           

          regards,

           

           

          Andrew Dinn

          1 of 1 people found this helpful
          • 2. Re: Errors in simple fault injection with TestNG and maven
            andou528

            Hi Andrew:


            Thank you very much for the reply. My intention is to try to inject the fault in different ways.

             

            Here, I create a testVariable as a member variable of the class which I initiate it in the constructor.

             

            I guess I should use the one you mentioned.

             

            targetLocation = "AFTER WRITE WebWriter.testVariable",
            
            

             

            However,  I still have the following error messages. Thank you very much.

            
            Setting org.jboss.byteman.allow.config.update=true
            Setting org.jboss.byteman.debug=true
            Setting org.jboss.byteman.verbose=true
            TransformListener() : accepting requests on localhost:9091
            BMUnit : loading text script = org.my.WebWriterTest3+testExceptionCatch
            TransformListener() : handling connection on port 9091
            -------- testExceptionCatch start ---------
            org.jboss.byteman.agent.Transformer : possible trigger for rule test exception catch in class org.my.app.WebWriter
            RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.my.app.WebWriter.openFile() java.io.PrintStream for rule test exception catch
            org.jboss.byteman.agent.Transformer : inserted trigger for test exception catch in class org.my.app.WebWriter
            Rule.execute called for test exception catch_0
            Rule.ensureTypeCheckedCompiled : error type checking rule test exception catch
            org.jboss.byteman.rule.exception.TypeException: ThrowExpression.typeCheck : exception type not declared by trigger method java.io.FileNotFoundException file org.my.WebWriterTest3+testExceptionCatch line 7
              at org.jboss.byteman.rule.expression.ThrowExpression.checkThrownTypeIsValid(ThrowExpression.java:426)
              at org.jboss.byteman.rule.expression.ThrowExpression.typeCheck(ThrowExpression.java:175)
              at org.jboss.byteman.rule.Action.typeCheck(Action.java:106)
              at org.jboss.byteman.rule.Rule.typeCheck(Rule.java:523)
              at org.jboss.byteman.rule.Rule.ensureTypeCheckedCompiled(Rule.java:449)
              at org.jboss.byteman.rule.Rule.execute(Rule.java:672)
              at org.jboss.byteman.rule.Rule.execute(Rule.java:653)
              at org.my.app.WebWriter.openFile(WebWriter.java:107)
              at org.my.WebWriterTest3.testExceptionCatch(WebWriterTest3.java:82)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:74)
              at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:176)
              at org.jboss.byteman.contrib.bmunit.BMNGAbstractRunner.run(BMNGAbstractRunner.java:56)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:189)
              at org.testng.internal.Invoker.invokeMethod(Invoker.java:666)
              at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:846)
              at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1170)
              at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
              at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
              at org.testng.TestRunner.runWorkers(TestRunner.java:1125)
              at org.testng.TestRunner.privateRun(TestRunner.java:749)
              at org.testng.TestRunner.run(TestRunner.java:600)
              at org.testng.SuiteRunner.runTest(SuiteRunner.java:317)
              at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:312)
              at org.testng.SuiteRunner.privateRun(SuiteRunner.java:274)
              at org.testng.SuiteRunner.run(SuiteRunner.java:223)
              at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
              at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
              at org.testng.TestNG.runSuitesSequentially(TestNG.java:995)
              at org.testng.TestNG.runSuitesLocally(TestNG.java:920)
              at org.testng.TestNG.run(TestNG.java:856)
              at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:62)
              at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:141)
              at org.apache.maven.surefire.Surefire.run(Surefire.java:177)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:345)
              at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1009)
            
            BMUnit : unloading text script = org.my.WebWriterTest3+testExceptionCatch
            TransformListener() : handling connection on port 9091
            retransforming org.my.app.WebWriter
            TransformListener() : handling connection on port 9091
            
            
            • 3. Re: Errors in simple fault injection with TestNG and maven
              adinn

              HI Andrew,

               

              Ok, I understand what is going wrong here and the problem is that your expectation of how THROW works is based on a misunderstanding. If we start by looking at the error message in the test output file we can begin to see the rotos of the mismatch:

               

              Rule.ensureTypeCheckedCompiled : error type checking rule test exception catch

              org.jboss.byteman.rule.exception.TypeException: ThrowExpression.typeCheck : exception type not declared by trigger method java.io.FileNotFoundException file org.my.WebWriterTest3+testExceptionCatch line 7

               

              The error is complaining that you are trying to throw an exception that is not declared by the trigger method i.e. by method WebWriter.openFile. That's because it is making sure that the exception can be thrown from th etrigger method to its caller. If you check the documentation for THROW (see Byteman Programmer’s Guide, 3.0.5, Apr 20, 2016 (Actions) and Byteman Programmer’s Guide, 3.0.5, Apr 20, 2016 (Rule Actions)) what it is trying to make clear is that when a rule executes a THROW is does not throw the exception from the injection point. What it actually does is throw an exception out of the triggering method to the caller. I guess the docs could be made a bit clearer but anyway that's why the check is being made.

               

              The reason Byteman does this is that the behaviour you are expecting, injecting a throw at an arbitrary point in a method, is full of pitfalls. For a start it makes it very difficult, in general, to determine whether or not an exception can propagate out of the method. When the trigger method does not declare the thrown exception (assuming it is nto an Error or RuntimException) then the consequence of such an escape is yet more unpredictable since it breaks the contract between the caller and called method. Even if the exception does not escape the new control flow which will result can be extremely hard to determine in some cases.

               

              So, when a rule THROW is executed Byteman short-circuits the rest of the method execution ensuring that the exceptional control flow routes the exception out of the method to the caller (n.b. it does not even execute code in finally blocks -- the one thing it does do is make sure that synchronized blocks are exited cleanly, releasing an other threads which may be waiting on the synchronized object). This makes it very clear what the effect of an injected throw will be (immediate exit from the method) and it makes it simple for Byteman to determine whether the throw meets or fails to meet the contract with the caller (it only legitimate if the method declares it as a checked exception or if it is an Error or RuntimeExcepiton).

               

              So, although your program includes a try catch block this will be ignored by Byteman when it checks the THROW. In your case the type checker has determined that your rule is invalid because WebWriter.openFile does not declare FileNotFoundException as a checked exception. In consequence your rule is rejected. If you modified the method so that it declared FileNotFoundException as a possible exception then Byteman would allow the rule to pass the type check and then the THROW would propagate the exception to your test method WebWriterTest3.testExceptionCatch. Of course, that's not really what you want either.

               

              In essence there is no way to ask Byteman to do what your test expects i.e. to have injected code generate an exception local to the trigger method and then catch it within that method. Not only is that not what THROW currently does it's also something I don't think Byteman can really support in a type-safe manner.

               

              regards,

               

               

              Andrew DInn