Agent failing to load sun.tools.attach.LinuxAttachProvider with OpenJDK 11 on Linux

[sannegrinovero]

Hi all,

when running our existing integration tests with:

openjdk version "11-ea" 2018-09-25

OpenJDK Runtime Environment 18.9 (build 11-ea+20)

OpenJDK 64-Bit Server VM 18.9 (build 11-ea+20, mixed mode)

On Fedora 28, the agent seems unable to attach. These same tests work fine on OpenJDK build 10.0.1+10 and various other JDKs I have a round for version 10, 9 and 8. (So we already learned to set -Djdk.attach.allowAttachSelf=true )

This is running Byteman version 4.0.3 :

byteman jar is /home/sanne/.m2/repository/org/jboss/byteman/byteman/4.0.3/byteman-4.0.3.jar
java.util.ServiceConfigurationError: com.sun.tools.attach.spi.AttachProvider: Provider sun.tools.attach.LinuxAttachProvider not found
     at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:588)
     at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1211)
     at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1220)
     at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1264)
     at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1299)
     at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1384)
     at jdk.attach/com.sun.tools.attach.spi.AttachProvider.providers(AttachProvider.java:258)
     at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:200)
     at org.jboss.byteman.agent.install.Install.attach(Install.java:451)
     at org.jboss.byteman.agent.install.Install.install(Install.java:161)
     at org.jboss.byteman.agent.install.Install.install(Install.java:116)
     at org.jboss.byteman.contrib.bmunit.BMUnitConfigState.loadAgent(BMUnitConfigState.java:516)
     at org.jboss.byteman.contrib.bmunit.BMUnitConfigState.pushConfigurationState(BMUnitConfigState.java:669)
     at org.jboss.byteman.contrib.bmunit.BMUnitRunner$1.evaluate(BMUnitRunner.java:73)
     ...      

Any suggestion of anything else I should try?

Thanks!

[adinn]

Hi Sanne,

Thanks for reporting this. I will investigate and get back to you.

However, ther eis another issue with using Byteman on JJDK11. Even if the attach were to work you might still face problems. Byteman 4.0.3 cannot currently consume class files generated with a  JDK11 class file version because that will only available once it is rebuilt with an updated version of ASM (6.2). So, 4.0.3 may work on JDK11 but only with classes compiled by JDK10 or lower.

I am preparing a new micro-release simply to upgrade ASM. I will look into your problem before releasing it.

regards,

Andrew Dinn

[sannegrinovero]

Thanks! Let me know if you need any help testing it. Also, would it be worth while to consider a Byteman release not shading ASM? That would give the option to users to experiment with overriding the dependency; sure it might break Byteman but it in some cases it would just work without you needing to re-spin a release just for a minor update.

[adinn]

sannegrinovero  wrote:

Thanks! Let me know if you need any help testing it.

Ok, will do.

sannegrinovero  wrote:

Also, would it be worth while to consider a Byteman release not shading ASM? That would give the option to users to experiment with overriding the dependency; sure it might break Byteman but it in some cases it would just work without you needing to re-spin a release just for a minor update.

There are two potential problems with unshading (the reasons why I use shaded code in the first place). Firstly, it will break rather unaccountably if Byteman is used in a deployment which is also using ASM but relying on an incompatible version. Secondly, if Byteman is run in the bootstrap (i.e the byteman agent jar is hoisted into the bootstrap classpath) then it will break rather unaccountably if users do not also hoist the ASM jar into the bootstrap.

The combination of both problems is too horrible to think about (imagine hoisting Byteman and ASM into the bootstrap of a JVM which has already loaded some ASM classes via the classpath.

Now I grant that noit all users wil face these issues and some might be able to grapple with and bypass them but the latter will be few (I know there is you at least but I don't think cloning tech is advanced enough that we need worry about many others). However, anyone who knows enough to play around with ASM safely can just as easily build their own Byteman release with whatever ASM they want if they really need to experiment. It's only takes git clone, emacs pom.xml[1] and mvn install.

So, that's a no then.

regards,

Andrew Dinn

[1] your editor-of-choice may (for reasons I cannot really fathom) vary

[adinn]

Hi Sanne,

With a few tweaks I managed to build a Byteman 4.0.4-SNAPSHOT release using ASM 6.2. It have just uploaded it to the Sonatype snapshots repo if you would like to try it.

I could not obtain jdk11-ea+20 but I tested it with jdk11-ea+21 and successfully installed byteman into a running program using bminstall and injected/executed/uninjected rules using bmsubmit. I also built the SNAPSHOT release using jdk11-ea+21. That exercises install from the java command line and self-hosting install.

So, I think jdk11-ea+21 will be working once I push 4.0.4. However, there are two more things to report.

1) If I try using 4.0.3 with jdk11-ea+21 then I can install successfully the agent using bminstall and submit rules using bmsubmit. However, rules do not appear to be injected. I have not debugged this but I have no doubt it is because the Byteman agent has problems dealing with jdk11 bytecode thanks to the ASM version being out of date.

2) The error reported by jdk11-ea+20 in your tests looks suspicious. The service loader which is trying to load a class to handle the attach prints out

  com.sun.tools.attach.spi.AttachProvider: Provider sun.tools.attach.LinuxAttachProvider not found

There is no LinuxAttachProvider in jdk11 (nor, indeed, in 10 or 9). However, it does exist in jdk8. Is it possible that your test config could have added a jdk8 tools.jar to your path? It certainly looks like you are picking up a jdk8 version of VMAttach.

regards,

Andrew Dinn

[sannegrinovero]

Thanks!  Byteman 4.0.4-SNAPSHOT does indeed attach correctly, and passes the tests.

Interesting point about having a wrong tools.jar around; I'll double check this later, maybe a side effect of it actually missing in recent JDK builds - don't know yet but I verified that indeed our build fails with 4.0.3 and works fine after your changes.

Byteman 4.0.4-SNAPSHOT passes all tests correctly in our main module with OpenJDK 11 b20 (able to both attach the agent and inject the expected rules); I can't say yet about all other modules we have as we need to fix several other dependencies before I can get to that, that will take much longer.

Sanne

[sannegrinovero]

P.S. even if we're running builds and runtim tests on JDK11 previews, we have --target set to 8. Maybe that helps explaining why rules were being injected fine after this fix?

[adinn]

Hi Sanne,

Thanks for checking the 4.0.4 snapshot. It's very odd that it doesn't run up against the LinuxAttachProvider problem while 4.0.3 does. I don't really see how that might happen but I'm not so concerned given that the snapshot is ok. I will push a new release soon.

regards,

Andrew Dinn

[sannegrinovero]

Thanks again Andrew, I see you released 4.0.4 and we're now upgrading all Hibernate projects! Quite a race to get all things working on JDK11 :)

[adinn]

Hi Sanne,

Glad you spotted the 4.0.4 release. I uploaded it a few days ago but only got round to officially announcing it today. Thanks for you help in testing it.

regards,

Andrew Dinn