. . .
Is this maybe because this initLibraryPaths() method gets called too early before Byteman agent can intercept this call? I haven't yet had a chance to dig deeper and it might be a while before I get a chance to come back and debug this more, so just checking here. I'm on OpenJDK 8u242b08.
I don't know for sure but I think this is extremely likely. If you switch on verbose trace and see logging saying the rule has been injected with no accompanying errors then I see no other reason why the injected would not get run. These are very simple rules which have been shown to work in many other injection contexts and so it is extremely unlikely that the injection would be failing silently.
Also, given how you describe its operation above I'd assume that the call from System has to happen before the Byteman agent jar can be loaded. This will not be th eonly such case. Not surprisingly, there are a whole host of things the JDK runtime needs to execute before an agent can safely be loaded,-- more before it can safely be started. All those actions are always going to be impossible to intercept via Byteman rules.
n.b. there is also a small set of methods that cannot be injected into even after loading Byteman. That's so because of the danger of entering a recursive triggering loop. Byteman does its best to detect recursive entry into injected code. If that happens from code which manages rule injection/execution it backs out of the injected trigger code. Clearly, the code which performs this check is unable to avoid such recursion. Byteman finesses this by blacklisting the methods used to perform the check. (essentially a ThreadLocal lookup).