Yes, as far as I know this ought to work. I can't really identify exactly what is going wrong here as you are using Oracles's JVM (so it seems) and I only have sources for the icedtea6 release builds which are currently based on release b39. However, there ought not to be any signiifcant difference in how they are implemented.
The error itself is not very informative since it merely indicates that something went wrong during the attempt to transform the class bytes for a given class i..e below the point where the JVMTI code invokes the agent. The error get sprinted when control returns to the JVMTI layer and it notices that an exception occurred. I have no reason to expect such an error so it might be useful to try to find out more info. There are two things you could do to help pin down what is going wrong.
Firstly, assuming it will not compromise the privacy of your app, could you show me the rules that are being loaded at startup. If that is not an option then could you try to make up some rules which manifest the same failure and then post those rules
Secondly, could you run with -Dorg.jboss.byteman.verbose added on the command line and show me any output that Byteman prints to the command line as a result of setting that property. If Byteman does produce some output then I may also need you to try to collect any transformed bytecode generated by Byteman but we'll negotiate that if and when we see what trace output is generated.
Meanwhile, I'll do some experiments see if I can generate a similar error. Thanks very much for reporting this issue and for any further info you can provide.