Bytecode not refreshed on runtime rule injection | java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletRequest

[garimakemwal]

I am trying to inject some code in Java classes ThreadPoolExecutor and Thread classes through a Helper class (attaching the Byteman Rules script).

As soon as the rules are loaded, the application starts to throw exceptions as follows, but behaves normally when the rules are unloaded.

Exception in thread "http-nio-8080-exec-2" Rule.execute called for runnable run_19java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletRequest

at java.lang.Class.getDeclaredMethods0(Native Method)

at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)

at java.lang.Class.privateGetPublicMethods(Class.java:2902)

at java.lang.Class.getMethods(Class.java:1615)

at org.jboss.byteman.rule.expression.MethodExpression.findMethod(MethodExpression.java:266)

at org.jboss.byteman.rule.expression.MethodExpression.typeCheck(MethodExpression.java:187)

at org.jboss.byteman.rule.Action.typeCheck(Action.java:106)

at org.jboss.byteman.rule.Rule.typeCheck(Rule.java:590)

at org.jboss.byteman.rule.Rule.ensureTypeCheckedCompiled(Rule.java:521)

at org.jboss.byteman.rule.Rule.execute(Rule.java:796)

at org.jboss.byteman.rule.Rule.execute(Rule.java:777)

at java.util.concurrent.ThreadPoolExecutor.beforeExecute(ThreadPoolExecutor.java)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Unknown Source)

NOTE :  This behaviour is intermittent, and works after repeated loading/unloading (Somewhat similar to this). The same set of rules work if I supply them while starting the application and not using any Custom Helper Class.

This is a spring boot application being started with following config :

java -Dorg.jboss.byteman.transform.all -javaagent:${BYTEMAN_HOME}/lib/byteman.jar=listener:true,boot:${BYTEMAN_HOME}/lib/byteman.jar -jar sample-byteman-logger-0.1.0.jar

Since the Helper class needs to inject code in packages java.util.concurrent and java.lang, I add the same in bootstrap classpath and then load the rules as follows with bmsubmit :

bmsubmit -b application-tracer-0.0.1-SNAPSHOT.jar [JAR containing helper class]

bmsubmit -l apprules.btm

The flag org.jboss.byteman.verbose gives me following debug logs, wasn't able to spot any errors while injection : 

org.jboss.byteman.agent.Transformer : possible trigger for rule runnable run in class org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run() void for rule runnable run

org.jboss.byteman.agent.Transformer : inserted trigger for runnable run in class org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller

org.jboss.byteman.agent.Transformer : possible trigger for rule runnable run in class org.LatencyUtils.SimplePauseDetector$SimplePauseDetectorThread

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.LatencyUtils.SimplePauseDetector$SimplePauseDetectorThread.run() void for rule runnable run

org.jboss.byteman.agent.Transformer : inserted trigger for runnable run in class org.LatencyUtils.SimplePauseDetector$SimplePauseDetectorThread

org.jboss.byteman.agent.Transformer : possible trigger for rule runnable run in class org.LatencyUtils.PauseDetector$PauseDetectorThread

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.LatencyUtils.PauseDetector$PauseDetectorThread.run() void for rule runnable run

org.jboss.byteman.agent.Transformer : inserted trigger for runnable run in class org.LatencyUtils.PauseDetector$PauseDetectorThread

org.jboss.byteman.agent.Transformer : possible trigger for rule runnable run in class org.springframework.boot.web.embedded.tomcat.TomcatWebServer$1

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.springframework.boot.web.embedded.tomcat.TomcatWebServer$1.run() void for rule runnable run

org.jboss.byteman.agent.Transformer : inserted trigger for runnable run in class org.springframework.boot.web.embedded.tomcat.TomcatWebServer$1

org.jboss.byteman.agent.Transformer : possible trigger for rule runnable run in class org.springframework.context.support.AbstractApplicationContext$1

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.springframework.context.support.AbstractApplicationContext$1.run() void for rule runnable run

org.jboss.byteman.agent.Transformer : inserted trigger for runnable run in class org.springframework.context.support.AbstractApplicationContext$1

org.jboss.byteman.agent.Transformer : possible trigger for rule before ThreadPoolExecutor in class java.util.concurrent.ThreadPoolExecutor

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into java.util.concurrent.ThreadPoolExecutor.beforeExecute(java.lang.Thread,java.lang.Runnable) void for rule before ThreadPoolExecutor

org.jboss.byteman.agent.Transformer : inserted trigger for before ThreadPoolExecutor in class java.util.concurrent.ThreadPoolExecutor

org.jboss.byteman.agent.Transformer : possible trigger for rule runnable run in class java.util.logging.LogManager$Cleaner

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into java.util.logging.LogManager$Cleaner.run() void for rule runnable run

org.jboss.byteman.agent.Transformer : inserted trigger for runnable run in class java.util.logging.LogManager$Cleaner

org.jboss.byteman.agent.Transformer : possible trigger for rule runnable run in class java.lang.ref.Finalizer$FinalizerThread

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into java.lang.ref.Finalizer$FinalizerThread.run() void for rule runnable run

org.jboss.byteman.agent.Transformer : inserted trigger for runnable run in class java.lang.ref.Finalizer$FinalizerThread

org.jboss.byteman.agent.Transformer : possible trigger for rule runnable run in class java.lang.ref.Reference$ReferenceHandler

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into java.lang.ref.Reference$ReferenceHandler.run() void for rule runnable run

org.jboss.byteman.agent.Transformer : inserted trigger for runnable run in class java.lang.ref.Reference$ReferenceHandler

org.jboss.byteman.agent.Transformer : possible trigger for rule runnable run in class java.lang.Thread

RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into java.lang.Thread.run() void for rule runnable run

org.jboss.byteman.agent.Transformer : inserted trigger for runnable run in class java.lang.Thread

Helper class :

public class LogHelper {

private static final Logger LOGGER = LoggerFactory.getLogger(LogHelper.class);
  public void executorDecorator() {
  LOGGER.info("Executor helper ");
  }
}

2 issues:

1. Do I need to restart the application to be able to see the bytecode changes, and why can't the application suddenly find 'javax/servlet/http/HttpServletRequest' in presence of simple rules. I don't see any errors for this also.

2. Is there a way to supply multiple JARs in boot: option of javagent so that I can inject the rules while starting the application (No such exceptions are seen with that) and add both byteman.jar and application-tracer  in bootstrap classpath simultaneously?

Something like : boot:[${BYTEMAN_HOME}/lib/byteman.jar,application-tracer-0.0.1-SNAPSHOT.jar]

Config used :

• Byteman version : 4.0.4
• java version "1.8.0_151"

Java(TM) SE Runtime Environment (build 1.8.0_151-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

Please let me know if I'm missing something here.

Thanks!

[adinn]

Hi Garima,

garimakemwal  wrote:

I am trying to inject some code in Java classes ThreadPoolExecutor and Thread classes through a Helper class (attaching the Byteman Rules script).

As soon as the rules are loaded, the application starts to throw exceptions as follows, but behaves normally when the rules are unloaded.

Exception in thread "http-nio-8080-exec-2" Rule.execute called for runnable run_19java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletRequest

. . .

1. Do I need to restart the application to be able to see the bytecode changes, and why can't the application suddenly find 'javax/servlet/http/HttpServletRequest' in presence of simple rules. I don't see any errors for this also.

No, you don't need to restart the server to see the bytecode changes. In fact, that is the exact opposite of what is needed.

Byteman only injects changes into a running server. If you start up the server again then you will need to reinstall Byteman and have it reinject your rules in order for the changes to appear. Byteman never changes the state of bytecode stored in class files on the disk.

I think the reason you are seeing these errors is a classloader issue. The rule which is giving the problem here injects into the run() method of class Thread or any of  its subclasses. So, let's assume that a specific target run() method belongs to some class XXXThread. Class XXXThread may be loaded by the system loader or the bootstrap loader -- itr doesn't really matter.

Byteman resolves the helper class by passing its name to loadClass() using the classloader of XXXThread as the required loader. Thta's ok, because your helper class, LogHelper, is in a jar added to the bootstrap path so it  is visible to Byteman whatever the loader of XXXThread happens to be (even if it was loaded by an ear or war loader it should eventually delegate to the system/bootstrap loaders).

In the exception trace you can see that Byteman is trying to type-check a method call, So, that must be the call to method executorDecorator() and Byteman knows this is a builtin method provided by your helper class. Byteman  calls getDeclaredMethods() to find all the methods declared by your helper class in order to look for one with a name and signature that matches the call. Constructing the list of Method objects for your helper class's declared methods involves calling into the JVM and resolving references from your helper to any classes mentioned as method parameter types. So, in particular, that includes resolving the arguments to method storeInbound() which takes an HttpServletRequest and HttpServletResponse as arguments. Now resolution of a class is always initiated relative to its classloader i.e. the JVM will look for these classes in the bootstrap loader (since your class is in the bootstrap). Neither of those classes is accessible from the bootstrap loader.

In other words, hoisting your helper class into the bootstrap path is not enough if your helper methods refer to classes that only exist in the system path. To make this work you would need to hoist the jar containing the HttpServletRequest and HttpServletResponse classes into the bootstrap. However, if they also refer to classes which are in system path jars then you need to hoist them as well and so on until all the classes you want are in the bootstrap.

garimakemwal  wrote:

2. Is there a way to supply multiple JARs in boot: option of javagent so that I can inject the rules while starting the application (No such exceptions are seen with that) and add both byteman.jar and application-tracer  in bootstrap classpath simultaneously?

Something like : boot:[${BYTEMAN_HOME}/lib/byteman.jar,application-tracer-0.0.1-SNAPSHOT.jar]

I'm not completely sure why you are not seeing exceptions when using -javaagent to load the Byteman agent. If you add the byteman jar to the bootstrap and the helper jar is only in the syspath then the agent will  fail to load the helper class when injecting into a bootstrap class like Thread or some subclass like XXXThread which is also in the bootstrap path. Of course, it will find your helper if injecting into a class YYYThread which belongs in the system path. Perhaps you are lucky and your rule mostly got injected into subclasses of Thread that belong to the bootstrap. I say mostly, because it will definitely have been injected into Thread.run. Also, I suspect there are subclasses of Thread used by the Executor library that were also tried. You may have missed any errors when these were tried if you did not set verbose trace.

Anyway, to answer your question: yes, you can add multiple jars during bootstrap. All you need to do is add multiple boot:/path/to/my.jar options, separated by a comma i.e.

-javaagent:${BYTEMAN_HOME}/lib/byteman.jar=boot:[${BYTEMAN_HOME}/lib/byteman.jar,boot:application-tracer-0.0.1-SNAPSHOT.jar

However, I don't think that is going to help you avoid errors when your rule gets injected into Thread or subclasses of Thread that reside in the bootstrap path.

regards,

Andrew Dinn

[ochaloup]

Hi Garima,

this sounds like class loading problem. Yet it's interesting it works when you restart the application several times. The boot:file.jar option let the jar being loaded by bootstrap class loader. It's mostly the same but different way to add Java_OPTS -Xbootclasspath/a:file.jar.

Usually classes are search in top-down order by classloaders. The classes loaded by boostrap will be touched consistently as the first choice (cl

ad 1. you should not need to restart your application. Byteman is capable to change your runtime on demand. I expect there is some classloading clash as the reason for the NoClassDefFound exception.

ad 2. if you want to define to load multiple jars by bootstrap classloader you can use the boot: option multiple times. This should work:

boot:${BYTEMAN_HOME}/lib/byteman.jar,boot:application-tracer-0.0.1-SNAPSHOT.jar

Cheers

Ondra

[garimakemwal]

Thanks for the prompt response Andrew!

Adding additional jars to bootstrap classloaders actually helped resolving those exceptions.

But now even a simple rule as follows is throwing exception.

RULE trace inbound in HttpServlet

CLASS javax.servlet.http.HttpServlet

METHOD service(HttpServletRequest, HttpServletResponse)

HELPER LogHelper

AT ENTRY

IF true

DO storeInbound($req)

ENDRULE

Exception :

org.jboss.byteman.agent.Transformer : inserted trigger for trace inbound in HttpServlet in class javax.servlet.http.HttpServlet

2018-08-20 23:05:33.527 ERROR 30263 --- [8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint   :

java.lang.IllegalAccessError: tried to access class org.apache.tomcat.util.net.SocketWrapperBase$1 from class org.apache.tomcat.util.net.SocketWrapperBase

at org.apache.tomcat.util.net.SocketWrapperBase.<clinit>(SocketWrapperBase.java:814) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]

at org.apache.tomcat.util.net.NioEndpoint$Poller.register(NioEndpoint.java:705) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]

at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:390) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]

at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:475) [tomcat-embed-core-8.5.31.jar!/:8.5.31]

at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]

2018-08-20 23:05:43.419 ERROR 30263 --- [Engine[Tomcat]]] org.apache.catalina.core.ContainerBase   : Exception invoking periodic operation:

java.lang.IllegalAccessError: tried to access class org.apache.catalina.webresources.Cache$EvictionOrder from class org.apache.catalina.webresources.Cache

at org.apache.catalina.webresources.Cache.backgroundProcess(Cache.java:183) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]

at org.apache.catalina.webresources.StandardRoot.backgroundProcess(StandardRoot.java:605) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]

at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5631) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]

at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1390) [tomcat-embed-core-8.5.31.jar!/:8.5.31]

at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1394) [tomcat-embed-core-8.5.31.jar!/:8.5.31]

at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1394) [tomcat-embed-core-8.5.31.jar!/:8.5.31]

at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1362) [tomcat-embed-core-8.5.31.jar!/:8.5.31]

at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]

The definition of the helper method is :

public void storeInbound(HttpServletRequest request) throws InterruptedException {

  MDC.put("id", UUID.randomUUID().toString());

  if (request.getHeader("headId") != null) {

  MDC.put("headId", request.getHeader("headId"));

  }

  }

Not sure why 'tried to access class org.apache.tomcat.util.net.SocketWrapperBase$1 from class org.apache.tomcat.util.net.SocketWrapperBase' is happening, happens after the rule is injected. The class 'SocketWrapperBase' is already in bootstrap path as well.

Any leads would be helpful.

Thanks!

[garimakemwal]

Thanks Ondra, I was able to add multiple jars to classpath with boot: option. Added follow up issues to Andrew's reply as well, please let me know if you can help. Thanks in advance.

[adinn]

Hi Garima,

garimakemwal  wrote:

Thanks for the prompt response Andrew!

Adding additional jars to bootstrap classloaders actually helped resolving those exceptions.

Ok, so that seems to confirm my diagnosis.

garimakemwal  wrote:

But now even a simple rule as follows is throwing exception.

RULE trace inbound in HttpServlet

CLASS javax.servlet.http.HttpServlet

METHOD service(HttpServletRequest, HttpServletResponse)

HELPER LogHelper

AT ENTRY

IF true

DO storeInbound($req)

ENDRULE

Exception :

org.jboss.byteman.agent.Transformer : inserted trigger for trace inbound in HttpServlet in class javax.servlet.http.HttpServlet

2018-08-20 23:05:33.527 ERROR 30263 --- [8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint   :

java.lang.IllegalAccessError: tried to access class org.apache.tomcat.util.net.SocketWrapperBase$1 from class org.apache.tomcat.util.net.SocketWrapperBase

at org.apache.tomcat.util.net.SocketWrapperBase.<clinit>(SocketWrapperBase.java:814) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]

at org.apache.tomcat.util.net.NioEndpoint$Poller.register(NioEndpoint.java:705) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]

at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:390) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]

at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:475) [tomcat-embed-core-8.5.31.jar!/:8.5.31]

at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]

  . . .

Not sure why 'tried to access class org.apache.tomcat.util.net.SocketWrapperBase$1 from class org.apache.tomcat.util.net.SocketWrapperBase' is happening, happens after the rule is injected. The class 'SocketWrapperBase' is already in bootstrap path as well.

Hmm, that is interesting.

It appears that SocketWrapperBase$1 is an auxiliary class generated by the javac compiler in response to this code:

  public static final CompletionCheck COMPLETE_WRITE = new CompletionCheck() {
    @Override
    public CompletionHandlerCall callHandler(CompletionState state, ByteBuffer[] buffers,
      int offset, int length) {
        for (int i = 0; i < length; i++) {
    . . .

n.b. the above listing is taken from the latest github version and the line numbers are not quite right so there may be something I am missing here.

I don't (yet) understand why you are  seeing an IllegalAccessError here. The auxiliary class must be located in the bootstrap path (it is normally packaged into the same jar as its parent class SocketWrapperBase) otherwise you would be seeing a ClassNotFoundException. An auxiliary like this one is explicitly defined so that it can provide access to its declaring class. Moving both the defining class and the auxiliary to the bootstrap path should not change the accessibility of any method of SocketWrapperBase$1 that is called from SocketWrapperBase.

Looking at dependencies from class SocketWrapperBase$1 to other classes, it will be an implementor of interface CompletionCheck and have Object as its super. So, its class pool should only reference ByteBuffer, CompletionCheck, CompletionHandlerCall and CompletionState. ByteBuffer is in the bootstrap path and the other three classes are interfaces or enums defined by SocketWrapperBase. So, there is no issue regarding recursive visibility of references here.

Looking into the jdk8 JVM code an exception with this format error message only comes from one place in jdk8. It can happen when running in the interpreter and resolving a class reference. The linkResolver code will throw the exception when the following call returns false:

  Reflection::verify_class_access(fref_klass, sel_klass, true)

where sel_klass is SocketWrapperBase$1 and ref_klass is SocketWrapperBase. So. that pins things down rather tightly. I traced through the code for Reflection::verify_class_access and I cannot see why this should fail to work simply because the two classes are now both located int he bootstrap loader path.

I'll keep looking into this but I actually think you are employing the wrong tactic here. Importing system path classes into the bootstrap path just in order to make injection of a generic rule into bootstrap classes work with a helper that depends on those system classes is not really the best strategy for dealing with the classloader problem that you face. I suggest you try another tack.

I notice that your rule for class Thread and ThreadExecutor doesn't actually need refer to any of the HttpServlet classes. So, you might finesse the class reference problem by using two helpers, one for the rules that may end up injecting into bootstrap path classes and another for rules that will only inject into system path classes. You can hoist the first helper jar into the bootstrap and leave the second one (which depends on system path classes) in the system path (use sys:/path/to/secondhelper.jar). Alternatively, stick with one helper + jar and rely on inline rule code for rules that may be injected into bootstrap classes.

Your two sets of rules/helpers can still share side effects. You can publish/access some side-effects using Byteman Helper class builtins. Alternatively, your bootstrap path helper can write static data which can be read by the system path helper, If you want to trsnafer data in the other directions i.e. allow the bootstrap path helper to see data created by the system path helper then either i) make the system path helper write values into pubic static fields of the bootstrap path helper or ii) use reflection to access system helper static fields from methods of the bootstrap helper class.

If you do this then the helper resident in the system path will be able to receive instances of the HttpServlet classes as arguments and will be able to call their methods access methods/data of objects accessible from them (or static methods/data of system path classes). The bootstrap path helper will only be able to link to classes that exist in the bootstrap path. If it wants to handle objects form classes in the system path it will have to do so using reflection.

Would you like to try that plan and get back to me if you have problems getting it to work or run into some issue with it?

regards,

Andrew Dinn

[garimakemwal]

Hey Andrew, Separation of jars does seems like a good approach to avoid hoisting all the dependent jars to bootstrap classpath. I was observing similar class definition exceptions even after adding system jars using 'sys:', probably expects me to add all classes in a cascaded fashion. I'm avoiding this for now by adding this jar as a maven dependency, that somehow adds all classes at the right classloader-level. Will also try your approach of sharing data as I explore more use cases. Thanks Garima

[adinn]

Ok, great -- and thanks very much for reporting the problem and  persevering through to this point. Problem reports are always really appreciated.

If you run into any other issues trying to write rules that allow the two different helper to communicate let me know and I will be happy to provide any advice I can.

regards,

Andrew Dinn