Hi folks,

I have been using Byteman in HDFS fault injection test for some time. However since our HDFS bytecode version is targeted to 1.7, I always hit "java.lang.VerifyError" when submitting a rule.

The steps I used Byteman in HDFS are as below:

1.Check the namenode process:

[gpadmin@smdw sbin]$ ps -ef | grep java

hdfs 117777 1 8 04:50 pts/0 00:00:07 /opt/jdk1.7.0_45/bin/java -Dproc_namenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/tmp/hadoop-2.2.0-gphd-3.1.0.0/logs -Dhadoop.log.file=hadoop-hdfs-namenode-smdw.log -Dhadoop.home.dir=/tmp/hadoop-2.2.0-gphd-3.1.0.0 -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/tmp/hadoop-2.2.0-gphd-3.1.0.0/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Xmx4096m -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Xmx4096m -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Xmx4096m -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode

2.bminstall.sh -b -Dorg.jboss.byteman.transform.all 117777

3.bmsubmit.sh -l rule.btm

Since HDFS bytecode version is changed, bmsubmit.sh always throw following error info.:

VerifyError during retransformation : some rules may not have been correctly injected or uninjected!

java.lang.VerifyError

at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)

at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)

at org.jboss.byteman.agent.Retransformer.installScript(Retransformer.java:136)

at org.jboss.byteman.agent.TransformListener.handleScripts(TransformListener.java:337)

at org.jboss.byteman.agent.TransformListener.loadScripts(TransformListener.java:260)

at org.jboss.byteman.agent.TransformListener.handleConnection(TransformListener.java:213)

at org.jboss.byteman.agent.TransformListener.run(TransformListener.java:146)

HDFS bytecode version change is located in hadoop-project/pom.xml:

<!-- ProtocolBuffer version, used to verify the protoc version and -->

<!-- define the protobuf JAR version -->

<protobuf.version>2.5.0</protobuf.version>

<protoc.path>${env.HADOOP_PROTOC_PATH}</protoc.path>

<zookeeper.version>3.4.5-gphd-2.1.0.0-SNAPSHOT</zookeeper.version>

+<javac.source>1.6</javac.source>

+<javac.target>1.7</javac.target>

...

<plugin>

   <groupId>org.apache.maven.plugins</groupId>

   <artifactId>maven-compiler-plugin</artifactId>

   <version>2.5.1</version>

   <configuration>

      -<source>1.6</source>

      -<target>1.6</target>

      +<source>${javac.source}</source>

      +<target>${javac.target}</target>

   </configuration>

</plugin>

I highly suspect that it's the compiler target change from 1.6 to 1.7, which blocks byteman to be used in HDFS.

Does someone have any idea on this?

My testing environment:

JAVA:

java version "1.7.0_45"

Java(TM) SE Runtime Environment (build 1.7.0_45-b18)

Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

OS:

Red Hat Enterprise Linux Server release 6.5

Byteman Version:

byteman-2.2.0

Thanks,

Zhenglin