4 Replies Latest reply on Mar 7, 2016 1:12 PM by adinn

    JBossModules plugin issue

    rachmato

      Hello

       

      I have used the JBossModules plugin for Byteman to allow rules to IMPORT a custom JBoss Modules module and its HELPER class to do some manipulations of Wildfly's JGroups and Infinispan classes. The rules work as expected, except in the case of one rule, which is throwing ClassNotFoundException on classes imported by the module. I'm kind of mystified, as it works for the other rules in the rule set:

       

      The rule set looks like this:

      ###########################################################
      # Rules for minitoring ThreadPoolExecutors
      ###########################################################
      IMPORT org.wildfly.byteman.rule.helper.threadpool
      HELPER org.wildfly.byteman.rule.helper.threadpool.PeriodicJGroupsThreadPoolHelper
      
      RULE TP_init
      CLASS org.jgroups.protocols.TP
      METHOD void init()
      AT EXIT
      IF TRUE
      DO
      # thread pools are created in TP.init
        System.out.println("[BYTEMAN]: Adding executors for TP: " + $0 + " with regular = " + $0.getDefaultThreadPool() + ", OOB = " + $0.getOOBThreadPool() + ", internal = " + $0.getInternalThreadPool());
        addTransportThreadPoolsToMap($0);
      ENDRULE
      
      RULE JGroups periodic helper hook
      CLASS org.wildfly.byteman.rule.helper.threadpool.JGroupsPeriodicHelper
      METHOD periodicTrigger
      AT ENTRY
      IF TRUE
      DO  
        System.out.println("[BYTEMAN-P] periodicTrigger called by thread");
        System.out.println(dumpStatsForExecutorInMap()) ;
      ENDRULE
      
      RULE TP_destroy
      CLASS org.jgroups.protocols.TP
      METHOD destroy
      AT ENTRY
      IF TRUE
      DO
        System.out.println("[BYTEMAN]: Calling destroy for TP: " + $0);
        removeTransportThreadPoolsFromMap($0)
      ENDRULE
      
      RULE RejectedExecutionException_throw_jgroups
      CLASS java.util.concurrent.RejectedExecutionException
      METHOD <init>
      AT ENTRY
      BIND 
        executor:ThreadPoolExecutor = $0;
      IF TRUE
      DO
        System.out.println("[BYTEMAN]: JGroups throwing RejectedExecutionException for executor " + $0);
      ENDRULE
      
      

       

      and the stack trace on the exception looks like this:

       

      [JBossINF]  13:37:39,783 ERROR [org.jgroups.protocols.UDP] (unicast receiver,ee,perf32) failed receiving packet: java.lang.NoClassDefFoundError: org/jgroups/protocols/TP
      [JBossINF] at java.lang.Class.getDeclaredMethods0(Native Method)
      [JBossINF] at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
      [JBossINF] at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
      [JBossINF] at java.lang.Class.getMethod0(Class.java:3018)
      [JBossINF] at java.lang.Class.getMethod(Class.java:1784)
      [JBossINF] at org.jboss.byteman.agent.HelperManager.lookupLifecycleMethod(HelperManager.java:319)
      [JBossINF] at org.jboss.byteman.agent.HelperManager.getDetails(HelperManager.java:286)
      [JBossINF] at org.jboss.byteman.agent.HelperManager.installed(HelperManager.java:65)
      [JBossINF] at org.jboss.byteman.rule.Rule.installed(Rule.java:915)
      [JBossINF] at org.jboss.byteman.rule.Rule.ensureTypeCheckedCompiled(Rule.java:490)
      [JBossINF] at org.jboss.byteman.rule.Rule.execute(Rule.java:705)
      [JBossINF] at org.jboss.byteman.rule.Rule.execute(Rule.java:686)
      [JBossINF] at java.util.concurrent.RejectedExecutionException.&lt;init>(RejectedExecutionException.java:65)
      [JBossINF] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
      [JBossINF] at org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:33)
      [JBossINF] at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
      [JBossINF] at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
      [JBossINF] at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1697)
      [JBossINF] at org.jgroups.protocols.TP.receive(TP.java:1630)
      [JBossINF] at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:691)
      [JBossINF] at java.lang.Thread.run(Thread.java:745)
      [JBossINF] Caused by: java.lang.ClassNotFoundException: org.jgroups.protocols.TP
      [JBossINF] at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      [JBossINF] at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      [JBossINF] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
      [JBossINF] at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      [JBossINF] ... 21 more
      [JBossINF] 
      

       

      The class is not found, however the other rules are triggered by that class and the module definition for the module org.wildfly.byteman.rule.helper.threadpool  includes the JGroups jar as an import:

       

      <module xmlns="urn:jboss:module:1.1" name="org.wildfly.byteman.rule.helper.threadpool">
          <resources>
              <resource-root path="${project.build.finalName}.jar"/>
          </resources>
          <dependencies>
          <module name="org.infinispan"/>
              <module name="org.jgroups"/>
              <module name="org.apache.log4j"/>
          </dependencies>
      </module>
      

      org.wildfly.byteman.rule.helper.threadpool

       

      Before I start digging, any ideas?

        • 1. Re: JBossModules plugin issue
          rachmato

          Hmm, I kicked off another run of the same job, but this time with verbose enabled, and the plugin seems to work this time:

           

          [JBossINF]  08:46:04,761 INFO [stdout] (unicast receiver,ee,perf32) Rule.execute called for RejectedExecutionException_throw_jgroups_1
          [JBossINF]  [0m [0m08:46:04,761 INFO [stdout] (unicast receiver,ee,perf32) RejectedExecutionException_throw_jgroups execute() [JBossINF]  [0m [0m08:46:04,761 INFO [stdout] (unicast receiver,ee,perf32) [BYTEMAN]: JGroups throwing RejectedExecutionException for executor java.util.concurrent.RejectedExecutionException: Task org.jgroups.protocols.TP$MyHandler@24d7fb05 rejected from java.util.concurrent.ThreadPoolExecutor@2d104b62[Running, pool size = 300, active threads = 0, queued tasks = 100, completed tasks = 3389117]
          

           

          with no ClassNotFoundExceptions appearing.

           

          I'll keep you posted ...

          • 2. Re: JBossModules plugin issue
            adinn

            Hi Richard,

            Richard Achmatowicz wrote:

              . . .

            The class is not found, however the other rules are triggered by that class and the module definition for the module org.wildfly.byteman.rule.helper.threadpool  includes the JGroups jar as an import:

             

            1. <module xmlns="urn:jboss:module:1.1" name="org.wildfly.byteman.rule.helper.threadpool"> 
            2.     <resources> 
            3.         <resource-root path="${project.build.finalName}.jar"/> 
            4.     </resources> 
            5.     <dependencies> 
            6.     <module name="org.infinispan"/> 
            7.         <module name="org.jgroups"/> 
            8.         <module name="org.apache.log4j"/> 
            9.     </dependencies> 
            10. </module> 

            org.wildfly.byteman.rule.helper.threadpool

             

            Before I start digging, any ideas?

             

            When you say that the other rules are triggered do you mean all of the other rules or just some of them. In particular, is the rule injected into method org.jgroups.protocols.TP.destroy getting triggered?

             

            Another thing to check is whether your helper class is available by any other path than the module. Might you mistakenly have added your helper jar to the system/bootstrap classpath?

             

            Meanwhile, I'll see if I can reproduce this using a simpler module setup.

            • 3. Re: JBossModules plugin issue
              rachmato

              Hi Andrew

               

              I *believe* that during this erroneous run, I may have had an extra sys: entry which added the helper jar to the Byteman system classpath. The reason being that in early stages of development, I had my Byteman settings set up for packaging the IMPORT/HELPER classes as a simple jar file and not as a JBoss Modules module. So your guess that the error is due to this may be correct. I'll try and confirm one way or the other, but in any case, the issue is no longer a problem for me.

              • 4. Re: JBossModules plugin issue
                adinn

                Hi Richard,

                Richard Achmatowicz wrote:

                 

                I *believe* that during this erroneous run, I may have had an extra sys: entry which added the helper jar to the Byteman system classpath. The reason being that in early stages of development, I had my Byteman settings set up for packaging the IMPORT/HELPER classes as a simple jar file and not as a JBoss Modules module. So your guess that the error is due to this may be correct. I'll try and confirm one way or the other, but in any case, the issue is no longer a problem for me.

                 

                Ok, that's good as I think it would make sense of the outcome.

                 

                I'll assume that things are ok until you let me know otherwise :-)

                 

                regards,

                 

                 

                Andrew Dinn