10 Replies Latest reply on Sep 25, 2018 3:51 AM by adinn

    NoClassDefFoundError from a class within the same package

    jam01

      Hello, I'm having an issue with instrumenting a particular framework using Byteman and I was hoping to get some guidance.

       

      The rule:

      RULE FilterChainBuilder.build
      CLASS ^org.glassfish.grizzly.filterchain.FilterChainBuilder
      METHOD FilterChain build()
      AT ENTRY
      BIND filterIdx: int = $0.indexOfType(org.glassfish.grizzly.http.HttpServerFilter.class)
      IF filterIdx != -1
      DO $0.add(filterIdx + 1, new io.opentracing.contrib.grizzly.TracingHttpServerFilter($0.get(filterIdx + 1), getTracer()));
        $0.remove(filterIdx + 2)
      ENDRULE

       

      The error thrown :

      org.jboss.byteman.agent.Transformer : possible trigger for rule FilterChainBuilder.build in class org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder
      RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder.build() org.glassfish.grizzly.filterchain.FilterChain for rule FilterChainBuilder.build
      org.jboss.byteman.agent.Transformer : inserted trigger for FilterChainBuilder.build in class org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder
      Rule.execute called for FilterChainBuilder.build_19
      Transformer.getClassChecker : unable to load bytecode for for class org.glassfish.grizzly.filterchain.Filter
      ERROR 2018-09-17 10:37:45,302 [WrapperListener_start_runner] org.mule.runtime.module.launcher.MuleContainer: A Fatal error has occurred while the server was running: org/glassfish/grizzly/filterchain/Filter
      java.lang.NoClassDefFoundError: org/glassfish/grizzly/filterchain/Filter
          at java.lang.ClassLoader.findBootstrapClass(Native Method) ~[?:1.8.0_181]
          at java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:1015) ~[?:1.8.0_181]
          at java.lang.ClassLoader.loadClass(ClassLoader.java:413) ~[?:1.8.0_181]
          at java.lang.ClassLoader.loadClass(ClassLoader.java:411) ~[?:1.8.0_181]
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_181]
          at java.lang.ClassLoader.loadClass(ClassLoader.java:411) ~[?:1.8.0_181]
          at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_181]
          at org.mule.runtime.module.artifact.api.classloader.FineGrainedControlClassLoader.findParentClass(FineGrainedControlClassLoader.java:137) ~[mule-module-artifact-4.1.2.jar:4.1.2]
          at org.mule.runtime.module.artifact.api.classloader.FineGrainedControlClassLoader.loadClass(FineGrainedControlClassLoader.java:91) ~[mule-module-artifact-4.1.2.jar:4.1.2]
          at java.lang.ClassLoader.loadClass(ClassLoader.java:411) ~[?:1.8.0_181]
          at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_181]
          at org.jboss.byteman.rule.type.TypeGroup.create(TypeGroup.java:151) ~[?:?]
          at org.jboss.byteman.rule.type.TypeGroup.create(TypeGroup.java:120) ~[?:?]
          at org.jboss.byteman.rule.expression.NewExpression.typeCheck(NewExpression.java:129) ~[?:?]
          at org.jboss.byteman.rule.expression.MethodExpression.findMethod(MethodExpression.java:297) ~[?:?]
          at org.jboss.byteman.rule.expression.MethodExpression.typeCheck(MethodExpression.java:187) ~[?:?]
          at org.jboss.byteman.rule.Action.typeCheck(Action.java:106) ~[?:?]
          at org.jboss.byteman.rule.Rule.typeCheck(Rule.java:590) ~[?:?]
          at org.jboss.byteman.rule.Rule.ensureTypeCheckedCompiled(Rule.java:521) ~[?:?]
          at org.jboss.byteman.rule.Rule.execute(Rule.java:796) ~[?:?]
          at org.jboss.byteman.rule.Rule.execute(Rule.java:777) ~[?:?]
          at org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder.build(FilterChainBuilder.java) ~[?:?]

       

      What's throwing me off is that this gets executed from FilterChainBuilder which should definitely have access to Filter. I should mention that Grizzly dependency should be provided by the application container and this might somehow affect the classloader accessibility. I think this is more or less related to https://developer.jboss.org/thread/278621 but I was not entirely sure what the solution there was or if it applies to this case.

       

      I'm fairly new to Byteman so I appreciate any insight or guidance here.

        • 1. Re: NoClassDefFoundError from a class within the same package
          adinn

          Hi Jose,

           

          This looks like it is a classloader issue. Your rule is being injected into a subclass of org.glassfish.grizzly.filterchain.FilterChainBuilder, let's call it T. The rule system attempts to type check the rule using the classloader of T (let's call that cl_T) to resolve any types mentioned in the rule. The error occurs when trying to typecheck the class mentioned in the new expression in the DO clause.

           

          As you can see in the backtrace the rule system has delegated to a MuleSoft classloader, FineGrainedControlClassLoader, to look up the class name in your new invocation. It seems from the backtrace that this FineGrainedControlClassLoader has delegated through a loader delegation chain and ended up asking first the system loader and then the bootstrap loader to resolve io.opentracing.contrib.grizzly.TracingHttpServerFilter but has not been able to find that class anywhere up the paretn delegation chain. Why this has happened really depends upon how your app is deployed. Class TracingHttpServerFilter may well not be visible form this specific injection point.

           

          If you run with verbose trace enabled (set system property org.jboss.byteman.verbose) then you should be able to see the rule being injected and executed, allowing you to work out on which class this type check is failing. At that point I suggest you look at how your classes are deployed and try to understand why class TracingHttpServerFilter is not visible from the classloader of the rule's target class.

           

          regards,

           

           

          Andrew Dinn

          • 2. Re: NoClassDefFoundError from a class within the same package
            jam01

            Hey Andrew, first I gotta say that from all the oss project forums I've happened to look into, you are the most active and ready to lend a hand developer I've seen.

             

            I have a couple of follow-up things:

            1. How to you know that it is `TracingHttpServerFilter` that is failing to resolve? From Line 07 I thought it was class Filter from Grizzly itself.
            2. I am running it on verbose enabled, Line 03 indicates that `inserted trigger for FilterChainBuilder.build in class org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder`
            3. This is my javaagent directive `-javaagent:~/opentracing-agent.jar=boot~/opentracing-agent.jar` which contains the Tracing implementations.

             

            Thanks.

             

            - Jose

            • 3. Re: NoClassDefFoundError from a class within the same package
              jam01

              I added Grizzly dependencies to my agent and things started working. Ran into an incompatibility issues but was able to find the right version. But so, why does this happen? How could I find out what's preventing the Helper from reaching the library provided by the container?

               

              - Jose

              • 4. Re: NoClassDefFoundError from a class within the same package
                adinn

                Hi Jose

                 

                jam01  wrote:

                 

                Hey Andrew, first I gotta say that from all the oss project forums I've happened to look into, you are the most active and ready to lend a hand developer I've seen.

                 

                Thanks very much -- I might even (anonymously) quote you on that! :-)

                 

                jam01  wrote:

                 

                I have a couple of follow-up things:
                1. How to you know that it is `TracingHttpServerFilter` that is failing to resolve? From Line 07 I thought it was class Filter from Grizzly itself.

                 

                 

                Well, now you gave explained the setup I see that this is indeed what is going wrong but not in the way I had assumed. So, let me revise what I originally said. The giveaway that TracingHttpServerFilter is being resolved is in this part of the exception backtrace:

                 

                    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_181]  
                    at org.jboss.byteman.rule.type.TypeGroup.create(TypeGroup.java:151) ~[?:?]  
                    at org.jboss.byteman.rule.type.TypeGroup.create(TypeGroup.java:120) ~[?:?]  
                    at org.jboss.byteman.rule.expression.NewExpression.typeCheck(NewExpression.java:129) ~[?:?]  
                    at org.jboss.byteman.rule.expression.MethodExpression.findMethod(MethodExpression.java:297) ~[?:?]  

                 

                This is the type checker executing a call to NewExpression.typeCheck and trying to resolve the name of the class which follows the new keyword. Your rule only includes one new operation for TracingHttpServerFilter. So, the lookup that fails has to start by looking for that class. I believe the exception occurs when the VM tries to link-resolve TracingHttpServerFilter. At that point it fails to find dependent class Filter . . .

                 

                jam01  wrote:

                 

                I am running it on verbose enabled, Line 03 indicates that `inserted trigger for FilterChainBuilder.build in class org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder`

                 

                Right. Your rule has been injected into this subclass of the target class mentioned in the rule. Byteman tries  to resolve the name TracingHttpServerFilter by asking the classloader of FilterChainBuilder$StatelessFilterChainBuilder. Looking at the exception backtrace this delegates to the system loader (I am assuming that the frame for the call to Launcher$AppClassLoader.loadClass is the point where the system class loader is entered) and then the bootstrap loader (the first call to ClassLoader.loadClass above the AppClassLoader.loadClass call). It seems that TracingHttpServerFilter is found in the bootstrap under that call. However, there is a call above that, also to ClassLoader.loadClass, which appears to be attempting to resolve class Filter. It leads to the exception. That makes sense when looking at the following  . . .

                 

                jam01  wrote:

                 

                This is my javaagent directive `-javaagent:~/opentracing-agent.jar=boot~/opentracing-agent.jar` which contains the Tracing implementations.

                 

                So, you are hoisting the opentracing-agent jar into the bootstrap path. I assume this is needed in order to inject tracing rules into methods of bootstrap classes. Unfortunately, the filter class you are using, TracingHttpServerFilter, appears to have been written on the assumption it is deployed in the system classpath along with all the other Grizzly classes. It expects to be able to link against Grizzly classes (well, Filter, at least)  but it won't be able to do that if it is loaded/linked by the bootstrap loader and the Grizzly jars are loaded/linked via the system loader. Is Filter a parent for TracingHttpServerFilter? Is TracingHttpServerFilter your own class?

                 

                You can try going down the path of adding the Grizzly jars to the bootstrap path to avoid this problem. However, that's potentially a rabbit hole. You may need to add jars Grizzly needs and then jars they need and so on.

                 

                An alternative way of fixing this would be to modify class TracingHttpServerFilter (or some variant of it that you provide) so that it executes whatever Grizzly code it needs to use reflectively. You can use ClassLoader,getSystemClassLoader().findClass() to load Grizzly classes and that will work even when TracingHttpServerFilter is deployed in the bootstrap. You can then use the reflection API to create instances and lookup/invoke the methods that TracingHttpServerFilter needs to run.

                 

                I hope that clarifies things enough to let you investigate alternative options here. If you can tell me a bit more about the provenance of TracingHttpServerFilter I may be able to provide a bit more help on how to follow this alternative path.

                 

                regards,

                 

                 

                Andrew Dinn

                 

                • 5. Re: NoClassDefFoundError from a class within the same package
                  jam01

                  Hey Andrew, thanks again for your input!

                   

                  I've changed the Rule a bit from what I first wrote. It's like this now

                  RULE FilterChainBuilder.build
                  CLASS ^org.glassfish.grizzly.filterchain.FilterChainBuilder
                  METHOD FilterChain build()
                  AT ENTRY
                  IF NOT callerEquals("io.opentracing.contrib.grizzly.TracedFilterChainBuilder.build", true, true, 0, 1)
                  DO io.opentracing.contrib.grizzly.TracedFilterChainBuilder.addFiltersTo($0.patternFilterChain, getTracer())
                  ENDRULE

                   

                  The static method called adds some filters to the chain to intercept and record http calls, the meat of the logic is

                  filterList.add(new TracingResponseHttpServerFilter(weakRequestMap, tracer1));
                  filterList.add(new TracingRequestHttpServerFilter(weakRequestMap, filterList.get(workerIdx), tracer1));
                  filterList.remove(workerIdx);

                   

                  Both of these new instances are written by me and extend org.glassfish.grizzly.filterchain.BaseFilter.

                   

                  Like I mentioned before when I removed `provided` qualifier from the grizzly dependency on my agent things started working. However that eventually led me to some compatibility issues with the version the container is using. So I started experimenting a little. I removed the other rules that deal with boot classloader classes (java.executors) and I'm testing just this Grizzly rule. When I load this rule with boot I see the following:

                   

                  org.jboss.byteman.agent.Transformer : possible trigger for rule FilterChainBuilder.build in class org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder
                  RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder.build() org.glassfish.grizzly.filterchain.FilterChain for rule FilterChainBuilder.build
                  org.jboss.byteman.agent.Transformer : inserted trigger for FilterChainBuilder.build in class org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder
                  Rule.execute called for FilterChainBuilder.build_0
                  Transformer.getClassChecker : unable to load bytecode for for class org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder
                  <-- Wrapper Stopped

                   

                  When I load the rule through system I see the following:

                  org.jboss.byteman.agent.Transformer : possible trigger for rule FilterChainBuilder.build in class org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder
                  RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder.build() org.glassfish.grizzly.filterchain.FilterChain for rule FilterChainBuilder.build
                  org.jboss.byteman.agent.Transformer : inserted trigger for FilterChainBuilder.build in class org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder
                  ERROR 2018-09-22 19:50:50,064 [WrapperListener_start_runner] org.mule.runtime.module.launcher.MuleContainer: A Fatal error has occurred while the server was running: org/jboss/byteman/rule/exception/EarlyReturnException
                  java.lang.NoClassDefFoundError: org/jboss/byteman/rule/exception/EarlyReturnException
                      at org.glassfish.grizzly.filterchain.FilterChainBuilder.stateless(FilterChainBuilder.java:61) ~[?:?]
                      at org.mule.service.http.impl.service.server.grizzly.GrizzlyServerManager.(GrizzlyServerManager.java:106) ~[?:?]
                      at org.mule.service.http.impl.service.server.HttpListenerConnectionManager.initialise(HttpListenerConnectionManager.java:80) ~[?:?]
                      at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:52) ~[mule-core-4.1.3.jar:4.1.3]
                      at org.mule.service.http.impl.service.HttpServiceImplementation.start(HttpServiceImplementation.java:95) ~[?:?]
                      at org.mule.runtime.module.service.internal.manager.MuleServiceManager.startServices(MuleServiceManager.java:90) ~[mule-module-service-4.1.3.jar:4.1.3]
                      at org.mule.runtime.module.service.internal.manager.MuleServiceManager.start(MuleServiceManager.java:67) ~[mule-module-service-4.1.3.jar:4.1.3]
                      at org.mule.runtime.module.launcher.MuleContainer.start(MuleContainer.java:231) [mule-module-launcher-4.1.3.jar:4.1.3]
                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
                      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_181]
                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
                      at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
                      at org.mule.runtime.module.reboot.internal.MuleContainerWrapper.start(MuleContainerWrapper.java:49) [mule-module-boot-ee-4.1.3.jar:4.1.3]
                      at org.tanukisoftware.wrapper.WrapperManager$11.run(WrapperManager.java:4330) [wrapper-3.5.35.jar:3.5.35]
                  Caused by: org.mule.runtime.module.artifact.api.classloader.exception.CompositeClassNotFoundException: Cannot load class 'org.jboss.byteman.rule.exception.EarlyReturnException': [
                      org.jboss.byteman.rule.exception.EarlyReturnException, 
                      Class 'org.jboss.byteman.rule.exception.EarlyReturnException' not found in classloader for artifact 'container'.]
                      at org.mule.runtime.module.artifact.api.classloader.FineGrainedControlClassLoader.loadClass(FineGrainedControlClassLoader.java:96) ~[mule-module-artifact-4.1.3.jar:4.1.3]
                      at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_181]
                      ... 14 more
                  Caused by: java.lang.ClassNotFoundException: org.jboss.byteman.rule.exception.EarlyReturnException
                      at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_181]
                      at org.mule.runtime.module.artifact.api.classloader.FineGrainedControlClassLoader.findLocalClass(FineGrainedControlClassLoader.java:167) ~[mule-module-artifact-4.1.3.jar:4.1.3]
                      at org.mule.runtime.module.artifact.api.classloader.FineGrainedControlClassLoader.loadClass(FineGrainedControlClassLoader.java:84) ~[mule-module-artifact-4.1.3.jar:4.1.3]
                      at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_181]
                      ... 14 more

                   

                  And when I load the rule through system and have boot point to byteman.jar I get:

                  org.jboss.byteman.agent.Transformer : possible trigger for rule FilterChainBuilder.build in class org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder
                  RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder.build() org.glassfish.grizzly.filterchain.FilterChain for rule FilterChainBuilder.build
                  org.jboss.byteman.agent.Transformer : inserted trigger for FilterChainBuilder.build in class org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder
                  Rule.execute called for FilterChainBuilder.build_0
                  Rule.ensureTypeCheckedCompiled : error type checking rule FilterChainBuilder.build
                  org.jboss.byteman.rule.exception.TypeException: Rule.typecheck : unknown helper class io.opentracing.contrib.agent.OpenTracingHelper for rule FilterChainBuilder.build
                      at org.jboss.byteman.rule.Rule.ensureHelperClass(Rule.java:1009)
                      at org.jboss.byteman.rule.Rule.typeCheck(Rule.java:568)
                      at org.jboss.byteman.rule.Rule.ensureTypeCheckedCompiled(Rule.java:521)
                      at org.jboss.byteman.rule.Rule.execute(Rule.java:796)
                      at org.jboss.byteman.rule.Rule.execute(Rule.java:777)
                      at org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder.build(FilterChainBuilder.java)
                      at org.mule.service.http.impl.service.server.grizzly.GrizzlyServerManager.(GrizzlyServerManager.java:132)
                      at org.mule.service.http.impl.service.server.HttpListenerConnectionManager.initialise(HttpListenerConnectionManager.java:80)
                      at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:52)
                      at org.mule.service.http.impl.service.HttpServiceImplementation.start(HttpServiceImplementation.java:95)
                      at org.mule.runtime.module.service.internal.manager.MuleServiceManager.startServices(MuleServiceManager.java:90)
                      at org.mule.runtime.module.service.internal.manager.MuleServiceManager.start(MuleServiceManager.java:67)
                      at org.mule.runtime.module.launcher.MuleContainer.start(MuleContainer.java:231)
                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                      at java.lang.reflect.Method.invoke(Method.java:498)
                      at org.mule.runtime.module.reboot.internal.MuleContainerWrapper.start(MuleContainerWrapper.java:49)
                      at org.tanukisoftware.wrapper.WrapperManager$11.run(WrapperManager.java:4330)

                   

                   

                  Sorry for the rambling. If it seems I'm just shooting in every direction it's because I am

                  I appreciate your input.

                   

                  - Jose

                  • 6. Re: NoClassDefFoundError from a class within the same package
                    jam01

                    I spent some time learning more about the classloading architecture for Mule and also about classloading in general. From that and other threads here I think what I need is to setup two agents. The first one would be a "boot" agent that'll house my universal Helper and all Rules and classes for instrumenting boot path classes. The second would be a "system" agent that'd house everything else. I've set it up through the maven assembly plugin, you can take a look at both agents here if that'd help https://github.com/jam01/java-mule-agent

                     

                    I felt this was the right approach, however... haha sadly I'm now getting the following error

                    Executor.execute execute()
                    org.jboss.byteman.agent.Transformer : possible trigger for rule FilterChainBuilder.build in class org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder
                    RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder.build() org.glassfish.grizzly.filterchain.FilterChain for rule FilterChainBuilder.build
                    org.jboss.byteman.agent.Transformer : inserted trigger for FilterChainBuilder.build in class org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder
                    Rule.execute called for FilterChainBuilder.build_10
                    Rule.ensureTypeCheckedCompiled : error type checking rule FilterChainBuilder.build
                    org.jboss.byteman.rule.exception.TypeException: MethodExpression.typeCheck : invalid path io.opentracing.contrib.grizzly.TracedFilterChainBuilder to static method addFiltersTo file jar:file:/home/jam01/Dev/IntelliJIdea/java-mule-agent/opentracing-mule-agent-system/target/opentracing-mule-agent-system.jar!/otarules.btm line 20
                        at org.jboss.byteman.rule.expression.MethodExpression.typeCheck(MethodExpression.java:135)
                        at org.jboss.byteman.rule.Action.typeCheck(Action.java:106)
                        at org.jboss.byteman.rule.Rule.typeCheck(Rule.java:590)
                        at org.jboss.byteman.rule.Rule.ensureTypeCheckedCompiled(Rule.java:521)
                        at org.jboss.byteman.rule.Rule.execute(Rule.java:796)
                        at org.jboss.byteman.rule.Rule.execute(Rule.java:777)
                        at org.glassfish.grizzly.filterchain.FilterChainBuilder$StatelessFilterChainBuilder.build(FilterChainBuilder.java)
                        at org.mule.service.http.impl.service.server.grizzly.GrizzlyServerManager.(GrizzlyServerManager.java:132)
                        at org.mule.service.http.impl.service.server.HttpListenerConnectionManager.initialise(HttpListenerConnectionManager.java:80)
                        at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:52)
                        at org.mule.service.http.impl.service.HttpServiceImplementation.start(HttpServiceImplementation.java:95)
                        at org.mule.runtime.module.service.internal.manager.MuleServiceManager.startServices(MuleServiceManager.java:90)
                        at org.mule.runtime.module.service.internal.manager.MuleServiceManager.start(MuleServiceManager.java:67)
                        at org.mule.runtime.module.launcher.MuleContainer.start(MuleContainer.java:231)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        at java.lang.reflect.Method.invoke(Method.java:498)
                        at org.mule.runtime.module.reboot.internal.MuleContainerWrapper.start(MuleContainerWrapper.java:49)
                        at org.tanukisoftware.wrapper.WrapperManager$11.run(WrapperManager.java:4330)

                     

                    Not entirely sure what this means.

                    • 7. Re: NoClassDefFoundError from a class within the same package
                      adinn

                      Hi Jose,

                       

                      Don't worry about rambling - this all makes sense to me and I am beginning to understand what the problem is that you need to deal with.

                       

                      Your last idea, which was to try to separate out the agent rules for bootstrap code from the agent rules for non-bootstrap code was actually a step in the right direction. However, you cannot use two separate agents to achieve what you want. If you try to load the Byteman agent twice only one agent will load. That's because the agent has to rely on static variables (don't ask, no really!) so, as they say in the movie Highlander, there can be only one. Even worse, if you add the Byteman jar to both the system and bootstrap classpaths then you risk having some Byteman classes loaded by the system loader and others by the bootstrap loader, causing very mysterious linkage errors and potentially breaking Byteman altogether.

                       

                      Let's start with what is the root problem here. What you are trying to do involves some different goals that are hard (but, with a little applied trickery, not impossible) to reconcile.

                       

                      1) You want to inject Byteman rule code into bootstrap path classes -- which, since they belong to the JDK,  don't have anything to do with Grizzly

                      2) You want to inject Byteman rule code into system path classes which belong to Grizzly

                      3) Under case 1 you want to call out to methods that belong to class TracingHttpServerFilter which need to resolve against Grizzly classes

                      4) Under case 2 you want to call out to methods that belong to class TracingHttpServerFilter which need to resolve against Grizzly classes

                       

                      You have a few ways to change the behaviour here. One is to deploy Byteman in the bootstrap path instead of the (default) system path. Another is to insert the jar containing  TracingHttpServerFilter and/or the Grizzly jars into the bootstrap path instead of just adding them to the system path.

                       

                      First, let's consider what happens when you deploy the agent in the system path and the jars containing TracingHttpServerFilter and Grizzly stay in tthe system path. This fails on goal 1. Let's assume your rule wants to inject code into method append of JDK runtime class java.util.List (it's of little or no importance which method it is in realit). The injected code needs to be surrounded with a try catch that captures three potential exceptions that are defined in one of the Byteman packages, EarlyReturnException, ThrowException and RuntimeException. So, these classes are referenced in the modified bytecode that the Byteman agent constructs and installs. When the new version of java.util.List.append() gets called you get a NoClassDefFoundError. Why? Well the Byteman exception classes are only visible to the system loader. The JVM will only resolve references from a given class via its loader. Since  java.util.List was loaded via the bootstrap loader the JVM asks it to look up EarlyReturnException and, of course, it is not visible.

                       

                      What happens if we boost the Byteman jar into the bootstrap classpath so it is is visible from the bootstrap loader but leave TracingHttpServerFilter and Grizzly in the system path. In that case when you inject a rule into java.util.List.append() the Byteman exception classes are  resolved ok. Also, let's assume you inject a rule into a method of a system path class like FilterChainBuilder$StatelessFilterChainBuilder. That also resolves the Byteman exception classes because the system loader delegates the lookup to the bootstrap loader. So, goals 1 and 2 are not prejudiced. However, what about goals 3 and 4?

                       

                      Goal 3 will not be met if you leave the jar containing TracingHttpServerFilter in the system path. The rule injected into java.util.List.append will fail. The target class for the rule is loaded via the bootstrap loader.  So, when Byteman tries to resolve the reference to  TracingHttpServerFilter via the bootstrap loader it will not be visible. n.b. it cannot simply pick some other loader to resolve classes mentioned in the rule because that might lead to type mismatches which could produce invalid bytecode.

                       

                      However, goal 4 will still be met if you leave the jar containing TracingHttpServerFilter in the system path. The rule injected into StatelessFilterChainBuilder will resolve fine. StatelessFilterChainBuilder is loaded via the system loader and so the reference to TracingHttpServerFilter will find a class local to that loader.

                       

                      Now, what happens if you try to fix goal 3 by promoting the jar containing TracingHttpServerFilter into the bootstrap path? Unfortunately, goals 3 and 4 will both fail for a different reason. The rule injected into java.util.List.append will have no problem finding TracingHttpServerFilter because it is now located in the bootstrap path. The rule injected into StatelessFilterChainBuilder will also be able to resolve the helper class since the lookup via the system loader will delegate to the bootstrap loader. However, TracingHttpServerFilter itself will fail to link properly when it is loaded. It references Grizzly classes that are only available in  the system path. When the bootstrap loader tries to link TracingHttpServerFilteryou get a NoClassDefFoundError again.

                       

                      Your last resort seems to be also  to boost the Grizzly jar into the bootstrap path. That does allow TracingHttpServerFilter to resolve against the Grizzly classes, However, when you run the app you will probably find that some of the Grizzly jars reference classes that belong to other libraries that are also only visible via the system path. If you only exercise code in a few Grizzly classes you might get  lucky and never try to load those libraries. However, you are on a slippery slope where you may end up having to boost the whole of Glassfish and your app into the bootstrap path. That way madness lies!!!

                       

                      Ok, what else can you do? Well, if you look at the 4 goals here we satisfied 3 of them with the setup where we had Byteman in the bootstrap path and the jar containing TracingHttpServerFilter in the system path along with the Girzzly jars and the app jars. You can inject rules into system or bootstrap classes and you can use your Grizzly helper when injecting into system path classes, including Grizzly's own classes like StatelessFilterChainBuilder. Only goal 3 is missing.

                       

                      So, let's see what we can do to fix this. First things first: is it broke? Do you actually need to invoke TracingHttpServerFilter and/or Grizzly functionality from rule injected into JDK code? If the answer is no then this setup should be ok. Just make sure that any rules which get injected into bootstrap classes don't attempt to refer to TracingHttpServerFilter or to Grizzly classes.

                       

                      Alternatively, let's assume you do need to use functionality that belongs to class TracingHttpServerFilter from rules that target bootstrap classes. In that case you need to create a wrapper class that invokes the required functionality but does so without causing link errors. How do you do that? Load the class yourself and use reflection!

                       

                      You will have to put the wrapper class into a jar that is inserted into the bootstrap path so your rules can see it. It will have to lookup class TracingHttpServerFilter explicitly via the system loader. Then it will have to find the methods it wants to call and invoke the required behaviour using reflection. So, you need something like this:

                       

                      class TracingHttpServerFilterWrapper
                      {
                        static Class targetClass;
                        static Constructor targetConstructor
                        Method targetMethod;
                        . . .
                        static {
                          try {
                            targetClass = ClassLoader.findSystemClass("TracingHttpServerFilterWrapper");
                          } catch (Exception e) {
                            System.out.println("Oops, should not happen! " + e);
                          }
                          try {
                            for (Constructor  constructor : targetClass.getDeclaredConstructors()) {
                               if (constructor.
                            }
                          } catch (Exception e) {
                            System.out.println("Oops, should not happen! " + e);
                          }
                        }
                        public Object createIndirectInstance(Type1 arg1, ... Typen argn)
                        {
                          try {
                            return targetClass.newInstance(arg1, ... argn);
                          } catch Exception e) {
                            System.out.println("Oops, should not happen! " + e);
                            return null;
                          }
                        }
                        public Object invokeTargetMethod(Object targetInstance, Type1 arg1, ... Typen argn)
                        {
                          try {
                            return targetMethod.invoke(targetInstance, arg1, ... argn);
                          } catch Exception e) {
                            System.out.println("Oops, should not happen! " + e);
                            return null;
                          }
                        }
                      }
                      • 8. Re: NoClassDefFoundError from a class within the same package
                        jam01

                        Andrew,

                         

                        I misspoke about the two agents. I have setup two jars, one with boot related rules/classes and another jar with the actual agent which instructs JVM to load the former jar from boot through the manifest.

                        <manifestEntries>
                          <Agent-Class>io.opentracing.contrib.agent.OpenTracingAgent</Agent-Class>
                          <Premain-Class>io.opentracing.contrib.agent.OpenTracingAgent</Premain-Class>
                          <Can-Redefine-Classes>true</Can-Redefine-Classes>
                          <Can-Retransform-Classes>true</Can-Retransform-Classes>
                          <Boot-Class-Path>${path.opentracing.mule-boot}/opentracing-mule-boot.jar<Boot-Class-Path> //classes and rules for java.executors, nothing to do with grizzly
                        </manifestEntries>
                        

                         

                        Also, goal number 3 in your list is not right, only TracingHttpServerFilter needs to access Grizzly classes. Again, I probably misspoke somewhere.

                         

                        So I assumed, and your last post confirms it I think, that the 2 jar setup would work. The reflective explanation/example is great because I think I'll still need to use this approach because of Mule's class loading isolation. However, this error is confusing me:

                        org.jboss.byteman.rule.exception.TypeException: MethodExpression.typeCheck : invalid path io.opentracing.contrib.grizzly.TracedFilterChainBuilder to static method addFiltersTo file jar:file:/home/jam01/Dev/IntelliJIdea/java-mule-agent/opentracing-mule-agent-system/target/opentracing-mule-agent-system.jar!/otarules.btm line 20

                         

                        See at this point I'd expect an error about TracingHttpServerFilter not finding Grizzly's classes, because even though TracingHttpServerFilter is on the system class loader, Grizzly is in a child loader. Is that unreachability what's causing this error? If it is, it's not evident to me. Any other hints are appreciated.

                         

                        Again, thanks for your input.

                        • 9. Re: NoClassDefFoundError from a class within the same package
                          jam01

                          Oh! TracingHttpServerFilter is not available at all in the System Class Loader, probably because at that level there's no Grizzly. So it just doesn't load the class? I tested this by creating a new class without any dependencies and it was available on the Sys CL. I'm not sure if that sounds right. But if that's the case this complicates things a whole lot more :s

                           

                          Edit: Yes, TracingHttpServerFilter is only available in any CL as long as Grizzly is also on the same CL.

                          • 10. Re: NoClassDefFoundError from a class within the same package
                            adinn

                            Hi Jose

                            jam01  wrote:

                             

                            Oh! TracingHttpServerFilter is not available at all in the System Class Loader, probably because at that level there's no Grizzly. So it just doesn't load the class? I tested this by creating a new class without any dependencies and it was available on the Sys CL. I'm not sure if that sounds right. But if that's the case this complicates things a whole lot more :s

                             

                            Edit: Yes, TracingHttpServerFilter is only available in any CL as long as Grizzly is also on the same CL.

                             

                            Hmm, so that provides you with a slightly bigger challenge :-). Here is an alternative suggestion as to how to get the targetClass and targetMethod fields in your wrapper class initialised:

                             

                            When you have a rule that is injected into Grizzly then you know it will be resolving classes using a loader which can see class TracingHttpServerFilter. So, let's assume that there is a class GrizzlyInit with method init() that gets called when Grizzly functionality gets initialized. The a rule like this can be used to initialize the wrapper class

                             

                            CLASS GrizzlyInit
                            METHOD init()
                            AT ENTRY
                            BIND initDone = flag("GrizzlyInited")
                            IF !initDone
                            DO TracingHttpServerFilterWrapper.initClass(io.opentracing.contrib.grizzly.TracingHttpServerFilter.class)
                            END RULE

                             

                            So, as soon as GrizzlyInit.init() gets called (i.e. Grizzly starts doing work) your rule will trigger. The call to Byteman built-in method flag() will set a flag labelled with String "GrizzlyInited". At the first triggering it will have the default value false. So the rule will fire the first time and call TracingHttpServerFilterWrapper.initClass, pasing a reference to the Class<?> instance for  TracingHttpServerFilterWrapper to your wrapper class. If init() gets called twice the flag will have value true so the rule will not fire again.

                             

                            Obviously, you will need to tweak the wrapper class for this to work. Firstly, the code in the  static init block needs to be moved into a static method called initClass so it can save the class and lookup the target method. Secondly, methods createIndirectInstance and invokeTargeMethod wil need to include a check to see whethe the targetClass and targetMethod fields are non-null. Obviously, the wrapper class can only redirect calls once initClass has been called.

                             

                            So, all you need to do is find some code that you know Grizzly is guaranteed to execute early on and inject this side-effect. I hope that gives you some idea of how you  might be able to do what you need.

                             

                            regards,

                             

                             

                            Andrew Dinn