Have you tried simply not giving any of the options and just giving the -Dcom.sun.management ones? In Java 9 there were changes to the JDK which should allow logging to bootstrap correctly.
FWIW -Xbootclasspath:/a does still work.
James R. Perkins
Very similar situation as what I have; but with a few subtle tweaks: JBOSS EAP 7.1x and JMXTranagent & JDK 1.9x.
I will say -Xbootclasspath:/a isn't a replacement for -Xbootclasspath:/p . Similar, but I think they are some slightly different applications for /a vs /p. And, in my case swapping -Xbootclasspath:/a for -Xbootclasspath:/p didn't work: I got the same class not found issues as if I didn't use a boot class path at all (while specifying a java agent).
The concept of a "patch-module" (new in JDK 1.9x) maybe in play (as a solution to this). But it is not clear that anything in the LogManager area of JBOSS/Wildfly is modularized ala JDK 1.9 (of for that matter if 'patch-module' necessitates a jigsaw style module in the first place).
In the OP, since it didn't appear to need an agent (like my case); it isn't clear why a bootclasspath is needed at all (or if /a would work).
Related, I found a note in the RH/EAP customer knowledge base; that pretty much says, RH/JBOSS isn't gonna bother with "certifying" JBOSS EAP 7.x under JDK 9 as JDK 10 is out -- and basically JDK 9 is dead wood already. I also took that to imply that even if RH/JBOSS does 'certify' an EAP version under, for example JDK 10, that doesn't imply anything about an EAP being re-engineered into modules or such.
Wildfly 12 has some 'noise' in its release notes about 'more JDK 9 support'; again I don't think that says anything about it being modularized or not.....
...I'd love to find out if others come up with a solution or workaround for this....