Is this something that bytman could support?
I think you will have to ask the IBM J9 team about that or -- if you are really desperate -- read the J9 manual.
Byteman relies upon the underlying host JVM/JDK to support the Java agent Attach API. The error message suggests that -- at least with the default configuration -- J9 does not support this API. It may be that J9 does support it but disables it by default. I don't really know the answer to that.
Ok, I looked online and the IBM docs appears to suggest that Java agents are supported from the command line. It also looks like they may require a property setting for dynamic loading using the Attach API. Did this work using the J9 JDK7 release? If so then this may represent a change from enabling agents by default to disabling them by default. If that is correct you could try running with this define on the command line
The JDK8 docs offer this setting as a way of enabling agent loads to proceed.