5 Replies Latest reply on Feb 19, 2016 9:21 AM by rachmato

    Invalid stack height compiler error

    rachmato

      Hello Byteman

       

      I'm using a set of rules to examine thread pool usage in Wildfly. The same ruleset compiles and executes on Wildfly when running locally, but fails to pass a type check when running  on a remote test server. Both are using JDK 8 to run the test program. I wonder if you could help me determine what the problem is.

       

      Here is the ruleset: I set up a counter associated with a collection of threadpools to allow me to display threadpool state every X method calls for methods which make use of the threadpool.

      ###########################################################
      #
      # Rules for minitoring ThreadPoolExecutors
      # - every 10th time execute() is called, spits out stats
      #
      ###########################################################
      RULE TP_up
      CLASS org.jgroups.protocols.TP
      METHOD up
      AT ENTRY
      BIND
        regularPoolSize:int = $this.getRegularPoolSize();
        regularPoolSizeActive:int = $this.getRegularPoolSizeActive();
        regularMessages:int = $this.getRegularMessages();
        regularQueueSize:int = $this.getRegularQueueSize();
        regularMaxQueueSize:int = $this.getRegularMaxQueueSize();
        internalPoolSize:int = $this.getInternalPoolSize();
        internalPoolSizeActive:int = $this.getInternalPoolSizeActive();
        internalMessages:int = $this.getInternalMessages();
        internalQueueSize:int = $this.getInternalQueueSize();
        internalMaxQueueSize:int = $this.getInternalMaxQueueSize();
        oobPoolSize:int = $this.getOOBPoolSize();
        oobPoolSizeActive:int = $this.getOOBPoolSizeActive();
        oobMessages:int = $this.getOOBMessages();
        oobQueueSize:int = $this.getOOBQueueSize();
        oobMaxQueueSize:int = $this.getOOBMaxQueueSize();
      IF (incrementCounter($0) > 0) AND (readCounter($0) % 10 == 0)
      DO
        System.out.println("[BYTEMAN]: Calling TP.up()");
        System.out.println("[BYTEMAN]: TP thread pools [pool, activePool, messages, queue, maxQueue]");
        System.out.println("[BYTEMAN]:   regular [" + regularPoolSize + ", " + regularPoolSizeActive  + ", " + regularMessages + ", " + regularQueueSize + ", " + regularMaxQueueSize + "]");
        System.out.println("[BYTEMAN]:   OOB [" + oobPoolSize + ", " + oobPoolSizeActive  + ", " + oobMessages + ", " + oobQueueSize + ", " + oobMaxQueueSize + "]");
        System.out.println("[BYTEMAN]:   internal [" + internalPoolSize + ", " + internalPoolSizeActive  + ", " + internalMessages + ", " + internalQueueSize + ", " + internalMaxQueueSize + "]");
      ENDRULE
      
      
      RULE TP_down
      CLASS org.jgroups.protocols.TP
      METHOD down
      AT ENTRY
      BIND
        regularPoolSize:int = $this.getRegularPoolSize();
        regularPoolSizeActive:int = $this.getRegularPoolSizeActive();
        regularMessages:int = $this.getRegularMessages();
        regularQueueSize:int = $this.getRegularQueueSize();
        regularMaxQueueSize:int = $this.getRegularMaxQueueSize();
        internalPoolSize:int = $this.getInternalPoolSize();
        internalPoolSizeActive:int = $this.getInternalPoolSizeActive();
        internalMessages:int = $this.getInternalMessages();
        internalQueueSize:int = $this.getInternalQueueSize();
        internalMaxQueueSize:int = $this.getInternalMaxQueueSize();
        oobPoolSize:int = $this.getOOBPoolSize();
        oobPoolSizeActive:int = $this.getOOBPoolSizeActive();
        oobMessages:int = $this.getOOBMessages();
        oobQueueSize:int = $this.getOOBQueueSize();
        oobMaxQueueSize:int = $this.getOOBMaxQueueSize();
      IF (incrementCounter($0) > 0) AND (readCounter($0) % 10 == 0)
      DO
        System.out.println("[BYTEMAN]: Calling TP.down()");
        System.out.println("[BYTEMAN]: TP thread pools [pool, activePool, messages, queue, maxQueue]");
        System.out.println("[BYTEMAN]:   regular [" + regularPoolSize + ", " + regularPoolSizeActive  + ", " + regularMessages + ", " + regularQueueSize + ", " + regularMaxQueueSize + "]");
        System.out.println("[BYTEMAN]:   OOB [" + oobPoolSize + ", " + oobPoolSizeActive  + ", " + oobMessages + ", " + oobQueueSize + ", " + oobMaxQueueSize + "]");
        System.out.println("[BYTEMAN]:   internal [" + internalPoolSize + ", " + internalPoolSizeActive  + ", " + internalMessages + ", " + internalQueueSize + ", " + internalMaxQueueSize + "]");
      ENDRULE
      
      

       

      RULE TP_init
      CLASS org.jgroups.protocols.TP
      METHOD <init>
      AT EXIT
      IF createCounter($0)
      DO
        System.out.println("[BYTEMAN]: Created counter for TP " + $0);
      ENDRULE
      
      
      RULE TP_destroy
      CLASS org.jgroups.protocols.TP
      METHOD destroy
      AT ENTRY
      IF deleteCounter($0)
      DO
        System.out.println("[BYTEMAN]: Calling destroy for TP " + $0);
      ENDRULE
      
      RULE RejectedExecutionException_throw_jgroups
      CLASS java.util.concurrent.RejectedExecutionException
      METHOD <init>
      AT ENTRY
      BIND
        executor:ThreadPoolExecutor = $0;
      IF TRUE
      DO
        System.out.println("[BYTEMAN]: Throwing RejectedExecutionException for executor " + $0);
      ENDRULE
      
      

       

      When I run these rules locally, the work fine, under TCP:

      17:30:14,651 INFO  [stdout] (MSC service thread 1-1) [BYTEMAN]: Created counter for TP TCP
      17:30:14,667 INFO  [stdout] (MSC service thread 1-1) [BYTEMAN]: Created counter for TP TCP
      17:30:15,063 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
      17:30:15,098 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) WFLYDS0013: Started FileSystemDeploymentService for directory /home/nrla/products/jboss-eap7-git-repo/testsuite/integration/clustering/target/wildfly-SYNC-tcp-0/standalone/deployments
      17:30:15,265 INFO  [org.wildfly.iiop.openjdk] (MSC service thread 1-4) WFLYIIOP0009: CORBA ORB Service started
      17:30:15,335 INFO  [org.jboss.ws.common.management] (MSC service thread 1-3) JBWS022052: Starting JBossWS 5.1.2.Final-redhat-2 (Apache CXF 3.1.4.redhat-1)
      17:30:18,314 INFO  [stdout] (MSC service thread 1-7) [BYTEMAN]: Calling TP.down()
      17:30:18,314 INFO  [stdout] (MSC service thread 1-7) [BYTEMAN]: TP thread pools [pool, activePool, messages, queue, maxQueue]
      17:30:18,314 INFO  [stdout] (MSC service thread 1-7) [BYTEMAN]:   regular [0, 0, 0, 0, 100]
      17:30:18,315 INFO  [stdout] (MSC service thread 1-7) [BYTEMAN]:   OOB [0, 0, 0, 0, 0]
      17:30:18,315 INFO  [stdout] (MSC service thread 1-7) [BYTEMAN]:   internal [1, 0, 0, 0, 100]
      
      

       

      and under UDP:

       

      19:40:36,344 INFO  [stdout] (MSC service thread 1-3) [BYTEMAN]: Created counter for TP UDP
      19:40:36,357 INFO  [stdout] (MSC service thread 1-3) [BYTEMAN]: Created counter for TP UDP
      19:40:36,424 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-7) WFLYDS0013: Started FileSystemDeploymentService for directory /home/nrla/products/jboss-eap7-git-repo/testsuite/integration/clustering/target/wildfly-SYNC-udp-1/standalone/deployments
      19:40:36,426 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "remote-failover-test.jar" (runtime-name: "remote-failover-test.jar")
       19:40:36,800 INFO  [stdout] (ViewHandler,ee,node-0) [BYTEMAN]: Calling TP.down()
      19:40:36,800 INFO  [stdout] (ViewHandler,ee,node-0) [BYTEMAN]: TP thread pools [pool, activePool, messages, queue, maxQueue]
      19:40:36,800 INFO  [stdout] (ViewHandler,ee,node-0) [BYTEMAN]:   regular [15, 0, 3, 0, 100]
      19:40:36,800 INFO  [stdout] (ViewHandler,ee,node-0) [BYTEMAN]:   OOB [20, 0, 423, 0, 0]
      19:40:36,800 INFO  [stdout] (ViewHandler,ee,node-0) [BYTEMAN]:   internal [2, 0, 16, 0, 100]
      
      

       

      When I run them remotely, under UDP only, I get the following exception:

      [

      JBossINF]  15:58:16,126 INFO [stdout] (MSC service thread 1-7) [BYTEMAN]: Created counter for TP UDP
      [JBossINF]  15:58:16,136 INFO [stdout] (MSC service thread 1-7) [BYTEMAN]: Created counter for TP UDP
      [JBossINF]  15:58:16,195 INFO [stdout] (MSC service thread 1-7) Rule.ensureTypeCheckedCompiled : error compiling rule TP_up
      [JBossINF]  15:58:16,196 INFO [stdout] (MSC service thread 1-7) org.jboss.byteman.rule.exception.CompileException: Event.compile : invalid stack height 3 expecting 0
      [JBossINF]  15:58:16,196 INFO [stdout] (MSC service thread 1-7) at org.jboss.byteman.rule.Event.compile(Event.java:311)
      [JBossINF]  15:58:16,196 INFO [stdout] (MSC service thread 1-7) at org.jboss.byteman.rule.compiler.Compiler.compileBytes(Compiler.java:401)
      [JBossINF]  15:58:16,196 INFO [stdout] (MSC service thread 1-7) at org.jboss.byteman.rule.compiler.Compiler.getHelperAdapter(Compiler.java:63) [JBossINF]  15:58:16,196 INFO [stdout] (MSC service thread 1-7) at org.jboss.byteman.rule.Rule.compile(Rule.java:578) 
      [JBossINF]  15:58:16,196 INFO [stdout] (MSC service thread 1-7) at org.jboss.byteman.rule.Rule.ensureTypeCheckedCompiled(Rule.java:488)
      [JBossINF]  15:58:16,196 INFO [stdout] (MSC service thread 1-7) at org.jboss.byteman.rule.Rule.execute(Rule.java:705)
      [JBossINF]  15:58:16,196 INFO [stdout] (MSC service thread 1-7) at org.jboss.byteman.rule.Rule.execute(Rule.java:686)
      [JBossINF]  15:58:16,196 INFO [stdout] (MSC service thread 1-7) at org.jgroups.protocols.TP.up(TP.java)
      [JBossINF]  15:58:16,196 INFO [stdout] (MSC service thread 1-7) at org.jgroups.protocols.TP.init(TP.java:1205)
      [JBossINF]  15:58:16,197 INFO [stdout] (MSC service thread 1-7) at org.jgroups.protocols.UDP.init(UDP.java:260)
      [JBossINF]  15:58:16,197 INFO [stdout] (MSC service thread 1-7) at org.jgroups.stack.ProtocolStack.initProtocolStack(ProtocolStack.java:860)
      [JBossINF]  15:58:16,197 INFO [stdout] (MSC service thread 1-7) at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:481)
      JBossINF]  15:58:16,197 INFO [stdout] (MSC service thread 1-7) at org.jgroups.JChannel.init(JChannel.java:853)
      [JBossINF]  15:58:16,197 INFO [stdout] (MSC service thread 1-7) at org.jgroups.JChannel.&lt;init>(JChannel.java:159)
      [JBossINF]  15:58:16,197 INFO [stdout] (MSC service thread 1-7) at org.jboss.as.clustering.jgroups.JChannelFactory$1.run(JChannelFactory.java:95)
      [JBossINF]  15:58:16,197 INFO [stdout] (MSC service thread 1-7) at org.jboss.as.clustering.jgroups.JChannelFactory$1.run(JChannelFactory.java:92)
      
      

       

      Any insights into this error? Run with verbose?

        • 1. Re: Invalid stack height compiler error
          rachmato
          [JBossINF]  [0m [0m21:35:56,180 INFO [stdout] (MSC service thread 1-2) Rule.execute called for TP_init_11
          [JBossINF]  [0m [0m21:35:56,184 INFO [stdout] (MSC service thread 1-2) HelperManager.install for helper class org.jboss.byteman.rule.helper.Helper
          [JBossINF]  [0m [0m21:35:56,185 INFO [stdout] (MSC service thread 1-2) calling activated() for helper class org.jboss.byteman.rule.helper.Helper
          [JBossINF]  [0m [0m21:35:56,185 INFO [stdout] (MSC service thread 1-2) Default helper activated
          [JBossINF]  [0m [0m21:35:56,185 INFO [stdout] (MSC service thread 1-2) calling installed(TP_init) for helper classorg.jboss.byteman.rule.helper.Helper
          [JBossINF]  [0m [0m21:35:56,185 INFO [stdout] (MSC service thread 1-2) Installed rule using default helper : TP_init
          [JBossINF]  [0m [0m21:35:56,192 INFO [stdout] (MSC service thread 1-2) TP_init execute()
          [JBossINF]  [0m [0m21:35:56,193 INFO [stdout] (MSC service thread 1-2) [BYTEMAN]: Created counter for TP UDP
          [JBossINF]  [0m [0m21:35:56,204 INFO [stdout] (MSC service thread 1-2) Rule.execute called for TP_init_11
          [JBossINF]  [0m [0m21:35:56,204 INFO [stdout] (MSC service thread 1-2) TP_init execute()
          [JBossINF]  [0m [0m21:35:56,204 INFO [stdout] (MSC service thread 1-2) [BYTEMAN]: Created counter for TP UDP
          [JBossINF]  [0m [0m21:35:56,256 INFO [stdout] (MSC service thread 1-2) Rule.execute called for TP_up_10
          [JBossINF]  [0m [0m21:35:56,259 INFO [stdout] (MSC service thread 1-2) Rule.ensureTypeCheckedCompiled : error compiling rule TP_up
          [JBossINF]  [0m [0m21:35:56,260 INFO [stdout] (MSC service thread 1-2) org.jboss.byteman.rule.exception.CompileException: Event.compile : invalid stack height 3 expecting 0
          [JBossINF]  [0m [0m21:35:56,260 INFO [stdout] (MSC service thread 1-2) at org.jboss.byteman.rule.Event.compile(Event.java:311)
          [JBossINF]  [0m [0m21:35:56,260 INFO [stdout] (MSC service thread 1-2) at org.jboss.byteman.rule.compiler.Compiler.compileBytes(Compiler.java:401)
          [JBossINF]  [0m [0m21:35:56,260 INFO [stdout] (MSC service thread 1-2) at org.jboss.byteman.rule.compiler.Compiler.getHelperAdapter(Compiler.java:63)
          [JBossINF]  [0m [0m21:35:56,260 INFO [stdout] (MSC service thread 1-2) at org.jboss.byteman.rule.Rule.compile(Rule.java:578)
          [JBossINF]  [0m [0m21:35:56,260 INFO [stdout] (MSC service thread 1-2) at org.jboss.byteman.rule.Rule.ensureTypeCheckedCompiled(Rule.java:488)
          [JBossINF]  [0m [0m21:35:56,260 INFO [stdout] (MSC service thread 1-2) at org.jboss.byteman.rule.Rule.execute(Rule.java:705)
          [JBossINF]  [0m [0m21:35:56,260 INFO [stdout] (MSC service thread 1-2) at org.jboss.byteman.rule.Rule.execute(Rule.java:686)
          [JBossINF]  [0m [0m21:35:56,260 INFO [stdout] (MSC service thread 1-2) at org.jgroups.protocols.TP.up(TP.java)
          [JBossINF]  [0m [0m21:35:56,261 INFO [stdout] (MSC service thread 1-2) at org.jgroups.protocols.TP.init(TP.java:1205)
          [JBossINF]  [0m [0m21:35:56,261 INFO [stdout] (MSC service thread 1-2) at org.jgroups.protocols.UDP.init(UDP.java:260)
          [JBossINF]  [0m [0m21:35:56,261 INFO [stdout] (MSC service thread 1-2) at org.jgroups.stack.ProtocolStack.initProtocolStack(ProtocolStack.java:860)
          [JBossINF]  [0m [0m21:35:56,261 INFO [stdout] (MSC service thread 1-2) at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:481)
          [JBossINF]  [0m [0m21:35:56,261 INFO [stdout] (MSC service thread 1-2) at org.jgroups.JChannel.init(JChannel.java:853)
          [JBossINF]  [0m [0m21:35:56,261 INFO [stdout] (MSC service thread 1-2) at org.jgroups.JChannel.&lt;init>(JChannel.java:159)
          [JBossINF]  [0m [0m21:35:56,261 INFO [stdout] (MSC service thread 1-2) at org.jboss.as.clustering.jgroups.JChannelFactory$1.run(JChannelFactory.java:95)
          [JBossINF]  [0m [0m21:35:56,261 INFO [stdout] (MSC service thread 1-2) at org.jboss.as.clustering.jgroups.JChannelFactory$1.run(JChannelFactory.java:92)
          [JBossINF]  [0m [0m21:35:56,262 INFO [stdout] (MSC service thread 1-2) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
          [JBossINF]  [0m [0m21:35:56,262 INFO [stdout] (MSC service thread 1-2) at org.jboss.as.clustering.jgroups.JChannelFactory.createChannel(JChannelFactory.java:98)
          [JBossINF]  [0m [0m21:35:56,262 INFO [stdout] (MSC service thread 1-2) at o
          
          • 2. Re: Invalid stack height compiler error
            adinn

            Hi Richard,

             

            Thanks for reporting this. I'll look into it as soon as I can.

             

            regards,

             

             

            Andrew Dinn

            • 3. Re: Invalid stack height compiler error
              rachmato

              Hi Andrew

               

              I returned to this problem and found that in the Byteman rule definition, that these lines:

               

              regularMessages:int = $this.getRegularMessages();
              OOBMessages:int = $this.getOOBMessages(); 
              internalMessages:int = $this.getInternalMessages();
              

               

              should have been these lines:

               

              regularMessages:long = $this.getRegularMessages();
              OOBMessages:long = $this.getRegularMessages(); 
              internalMessages:long = $this.getRegularMessages();
              

               

              In other words, TP.getRegularMessages() returns a long, not an int and, in the incorrect bindings, I was implicitly casting from long to int. Same for the other two.

               

              Changing the binding definitions made the problem go away.

              However, its odd that it worked locally and not within our remotely executed SmartFrog jobs.

              Is it expected that this sort of cast will work, and only result in lost data? In other words, the stack height error should not be appearing?

              • 4. Re: Invalid stack height compiler error
                adinn

                Ah, I was wondering how this error could arise. I certainly wasn't able to reproduce it.

                 

                The disparity between int and long certainly chimes with the stack height being left at 3 (3 initializations times one slot difference). It looks as if what is happening here is that Byteman is inserting an l2i to convert the long return value to an int but is neglecting to account for the change in stack height associated with this operation.  The compiler uses a few utility routines to do things like numeric interconversions, box/unbox operations and automatic toString() transformations and these are meant to adjust the stack heights in the CompileContext as they do so (they also monitor the stack high-watermark so that we can post the max stack height at end of method generation). I'll check to see if there is a missing case there.

                 

                I also wondered if you may have only seen it on one machine because you only had compilation to bytecode enabled on that one machine. Is that a possibility?

                 

                regards,

                 

                 

                Andrew Dinn

                • 5. Re: Invalid stack height compiler error
                  rachmato

                  Yes, that is the case. The remote job has the -Dorg.jboss.byteman.compile.to.bytecode system property set and the local job does not.