Transformation failed after moving to compilation -target 1.7

[sannegrinovero]

Hi all,

I'm in the process of moving the build of Hibernate Search to Java7. After changing the compilation settings to output classes in the Java 7 format, some of our unit tests using Byteman fail.

It's very easy to reproduce, just checkout my branch BytemanErrorJava7 from Sanne/hibernate-search · GitHub and run "mvn install".

Or from an IDE, just run the JUnit test org.hibernate.search.test.configuration.integration.HibernateSearchSessionFactoryObserverTest , from the orm module, which uses @BMRules.

This is the relevant output:

byteman jar is /home/sanne/.m2/repository/org/jboss/byteman/byteman/2.1.4/byteman-2.1.4.jar
23:44:17,724 (main)  WARN DriverManagerConnectionProviderImpl:93 - HHH000402: Using Hibernate built-in connection pool (not for production use!)
org.jboss.byteman.agent.Transformer : Saving transformed bytes to ./org/hibernate/internal/SessionFactoryImpl.class
23:44:18,135 (main)  INFO Version:43 - HSEARCH000034: Hibernate Search [WORKING]
org.jboss.byteman.agent.Transformer : Saving transformed bytes to ./org/hibernate/search/spi/SearchFactoryBuilder.class
Exception in thread "Thread-0" java.lang.InternalError
  at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
  at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)
  at org.jboss.byteman.agent.Retransformer.removeScripts(Retransformer.java:292)
  at org.jboss.byteman.agent.TransformListener.handleScripts(TransformListener.java:335)
  at org.jboss.byteman.agent.TransformListener.deleteScripts(TransformListener.java:290)
  at org.jboss.byteman.agent.TransformListener.handleConnection(TransformListener.java:215)
  at org.jboss.byteman.agent.TransformListener.run(TransformListener.java:146)

I believe the problem might be the fact that ASM v.3 doesn't support Java7 classes?

I've tried to upgrade Byteman to use ASM 5.0.1.

Since I'm not familiar with the internals of Byteman I was hoping that maybe it would just work; I had to apply a couple of API adjustments and got it to compile, but I wasn't lucky as it fails the Byteman testsuite. I didn't find and ASM update guide, and it's all looking a bit intimidating to figure out on my own .

In case it's interesting my (broken) patch for Byteman to upgrade ASM is on Github too: branch Java7 of Sanne/byteman · GitHub .

I'm attaching the .class file mentioned at line 05 of the above log.

[adinn]

Hi Sanne,

ASM 3 does include support for transforming Java7 classes.

I have seen problems in the past when transforming Java7 code. However, they have not been to do with the transformation performed by Byteman -- they appeared to be related to the use of try with resources. In this case the byetcode in the class file includes what looks like a valid Byteman agent transformation -- it starts with an AT ENTRY injected call for rule "Factory build prohibitor".

  public org.hibernate.search.engine.spi.SearchFactoryImplementor buildSearchFactory();
    flags: ACC_PUBLIC
    Code:
      stack=3, locals=2, args_size=1
         0: ldc           #69                 // String Factory build prohibitor_1
         2: aload_0      
         3: aconst_null  
         4: invokestatic  #75                 // Method org/jboss/byteman/rule/Rule.execute:(Ljava/lang/String;Ljava/lang/Object;[Ljava/lang/Object;)V
         7: aload_0      
         8: getfield      #52                 // Field rootFactory:Lorg/hibernate/search/impl/MutableSearchFactory;

The exception handling code also looks good

        50: areturn      
        51: invokevirtual #104                // Method org/jboss/byteman/rule/exception/EarlyReturnException.getReturnValue:()Ljava/lang/Object;
        54: checkcast     #106                // class org/hibernate/search/engine/spi/SearchFactoryImplementor
        57: areturn      
        58: invokevirtual #110                // Method org/jboss/byteman/rule/exception/ThrowException.getThrowable:()Ljava/lang/Throwable;
        61: athrow       
        62: athrow       
      Exception table:
         from    to  target type
             0     7    51   Class org/jboss/byteman/rule/exception/EarlyReturnException
             0     7    58   Class org/jboss/byteman/rule/exception/ThrowException
             0     7    62   Class org/jboss/byteman/rule/exception/ExecuteException

That's all just the usual boiler plate.

The stack map table also seems to have been generated ok

      StackMapTable: number_of_entries = 6
           frame_type = 36 /* same */
           frame_type = 7 /* same */
           frame_type = 252 /* append */
             offset_delta = 4
        locals = [ class org/hibernate/search/engine/spi/SearchFactoryImplementor ]
           frame_type = 255 /* full_frame */
          offset_delta = 1
          locals = [ class org/hibernate/search/spi/SearchFactoryBuilder ]
          stack = [ class org/jboss/byteman/rule/exception/EarlyReturnException ]
           frame_type = 70 /* same_locals_1_stack_item */
          stack = [ class org/jboss/byteman/rule/exception/ThrowException ]
           frame_type = 67 /* same_locals_1_stack_item */
          stack = [ class org/jboss/byteman/rule/exception/ExecuteException ]

We get 3 extra frames for the 3 handlers added by Byteman all of which look ok to me.

I am not really sure what is wrong with the transformed bytecode. The rest of the file is quite small and I cannot see any obvious problems in the disassembly. I think I will have use gdb to try to catch the problem in the verifier code.

[sannegrinovero]

Hi Andrew,

thanks a lot, support very appreciated.

you're probably right that ASM3 generally works with Java7 as Byteman is still working fine for us in many other tests. The following link is what led me to believe it's not fully compatible though, as they mention "full support of Java7" as a new feature since 4.0 CR1 : http://asm.ow2.org/history.html. I wish there where more details on what exactly was fixed.

I tried to workaround the issue by changing the rule in various ways, but it consistently fails. It's failing even if I change the rule action to a single "debug(message)" statement, so I think it might be related to something unusual in the class it's instrumenting. I tried a quick an dirty refactoring the remote the only thing in there which I would consider less common: an inner class; but it keeps failing. As an experiment I even removed loggers & other static stuff, it still fails so I'm a bit clueless now. To be fair it's an ancient class which hasn't had to be touched in ages (no need so far), so I'll take it as an opportunity to reorganize some very fishy code, but if you could take a look with the debugger you probably want the messy edition ;-) Anything else I could do?

Sanne

[adinn]

Sanne Grinovero wrote:

Hi Andrew,

thanks a lot, support very appreciated.

Bug report equally as appreciated :-)

Sanne Grinovero wrote:


you're probably right that ASM3 generally works with Java7 as Byteman is still working fine for us in many other tests. The following link is what led me to believe it's not fully compatible though, as they mention "full support of Java7" as a new feature since 4.0 CR1 : http://asm.ow2.org/history.html. I wish there where more details on what exactly was fixed.

Hmm, "full support"? Well, maybe the support in 3.x is known to be flaky and unfixable. It certainly looks like it.

Sanne Grinovero wrote:

I tried to workaround the issue by changing the rule in various ways, but it consistently fails. It's failing even if I change the rule action to a single "debug(message)" statement, so I think it might be related to something unusual in the class it's instrumenting. I tried a quick an dirty refactoring the remote the only thing in there which I would consider less common: an inner class; but it keeps failing. As an experiment I even removed loggers & other static stuff, it still fails so I'm a bit clueless now. To be fair it's an ancient class which hasn't had to be touched in ages (no need so far), so I'll take it as an opportunity to reorganize some very fishy code, but if you could take a look with the debugger you probably want the messy edition ;-) Anything else I could do?

Yes, I think it is to do with the class since the Byteman modifications are trivial and just like many other cases which work. It may be something trivial like a bad index into the class pool or some bad data in the file. The bytecode is fairly small and I couldn't see anything obviously wrong with the method code. I think the only real way to pin this down is to step trough the verifier code. I'll try to schedule that in at some point but I don't have much time to spare just now.

[hardy.ferentschik]

Did we ever find out what causes the problem. I just lost parts of my sanity trying to write a new Byteman test which fails in the same manner. Of course I forgot abut https://hibernate.atlassian.net/browse/HSEARCH-1600 and Sanne mentioning this issue :-(

[hardy.ferentschik]

Hmm, seems like upgrading to the latest Byteman version (2.2.1) seems to do the trick.

[adinn]

Hi Hardy,

I have just released Byteman 3.0.0 which runs on ASM 5.0.3 and has better support for various new in JDK7/8/9 constructs than the one I was previously using (ASM 3.3.1). I hope the problems you and Sanne were facing will disappear with this release.

regards,

Andrew Dinn

[sannegrinovero]

Thanks Andrew! the ASM update sounds great.

I've updated Hibernate Search, all works fine (no regressions) but that troublesome test which we originally had starting this thread had to be adjusted to a different implementation so it no longer applies (i.e. I can't confirm it would have solved the problem, although it sounds likely).