0 Replies Latest reply on Jul 7, 2014 10:25 AM by zhenglin tao

    java.lang.VerifyError when using byteman in latest HDFS 2.2.0

    zhenglin tao Newbie

      Hi folks,

       

      I have been using Byteman in HDFS fault injection test for some time. However since our HDFS bytecode version is targeted to 1.7, I always hit "java.lang.VerifyError" when submitting a rule.

       

      The steps I used Byteman in HDFS are as below:

      1.Check the namenode process:

      [gpadmin@smdw sbin]$ ps -ef | grep java

      hdfs 117777 1 8 04:50 pts/0 00:00:07 /opt/jdk1.7.0_45/bin/java -Dproc_namenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/tmp/hadoop-2.2.0-gphd-3.1.0.0/logs -Dhadoop.log.file=hadoop-hdfs-namenode-smdw.log -Dhadoop.home.dir=/tmp/hadoop-2.2.0-gphd-3.1.0.0 -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/tmp/hadoop-2.2.0-gphd-3.1.0.0/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Xmx4096m -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Xmx4096m -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Xmx4096m -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode

      2.bminstall.sh -b -Dorg.jboss.byteman.transform.all 117777

      3.bmsubmit.sh -l rule.btm

      Since HDFS bytecode version is changed, bmsubmit.sh always throw following error info.:

      VerifyError during retransformation : some rules may not have been correctly injected or uninjected!

      java.lang.VerifyError

      at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)

      at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)

      at org.jboss.byteman.agent.Retransformer.installScript(Retransformer.java:136)

      at org.jboss.byteman.agent.TransformListener.handleScripts(TransformListener.java:337)

      at org.jboss.byteman.agent.TransformListener.loadScripts(TransformListener.java:260)

      at org.jboss.byteman.agent.TransformListener.handleConnection(TransformListener.java:213)

      at org.jboss.byteman.agent.TransformListener.run(TransformListener.java:146)

       

      HDFS bytecode version change is located in hadoop-project/pom.xml:

      <!-- ProtocolBuffer version, used to verify the protoc version and -->

      <!-- define the protobuf JAR version -->

      <protobuf.version>2.5.0</protobuf.version>

      <protoc.path>${env.HADOOP_PROTOC_PATH}</protoc.path>

      <zookeeper.version>3.4.5-gphd-2.1.0.0-SNAPSHOT</zookeeper.version>

      +<javac.source>1.6</javac.source>

      +<javac.target>1.7</javac.target>

      ...

      <plugin>

         <groupId>org.apache.maven.plugins</groupId>

         <artifactId>maven-compiler-plugin</artifactId>

         <version>2.5.1</version>

         <configuration>

            -<source>1.6</source>

            -<target>1.6</target>

            +<source>${javac.source}</source>

            +<target>${javac.target}</target>

         </configuration>

      </plugin>

       

      I highly suspect that it's the compiler target change from 1.6 to 1.7, which blocks byteman to be used in HDFS.

       

      Does someone have any idea on this?

       

      My testing environment:

      JAVA:

      java version "1.7.0_45"

      Java(TM) SE Runtime Environment (build 1.7.0_45-b18)

      Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

       

      OS:

      Red Hat Enterprise Linux Server release 6.5

       

      Byteman Version:

      byteman-2.2.0

       

       

      Thanks,

      Zhenglin