Another interesting data point. If I write a class whose main function does an InetAddress.getByName and then invokes org.jboss.modules.Main.main(String args), I can get my custom DNS resolver to be used. This implies JBoss is doing something early on in the process to explicitly exclude my resolver. Still have no clue why though.
It's a bit tricky, but I think doable. Looking at the InetAddress class, it does the nameservers initialization in a static block of that class and uses Thread context classloader to find any custom defined nameservers. So what this means is you will have to do something like this on a fresh install of the WildFly/JBoss EAP server:
1. Create a JBoss module for your custom nameserver provider. There's documentation on how to do this in relevant server/JBoss modules docs. If you don't find it, please ask either in this thread or a new one. Assume you name that module "com.mycompany.dnsprovider".
2. Edit the module.xml of org.jboss.modules to add a dependency on your custom module *and* set the services=import attribute to true. Something like:
<module name="com.mycompany.dnsprovider" services="import"/>
3. In the stanadlone.conf (or relevant startup config file) pass the sun.net.spi.nameservice.provider.n system property in the JAVA_OPTS section:
I think that should be it.
P.S: It's still not 100% fool proof but yet will "almost always" work because I think you can probably expect the InetAddress class to be accessed very early in the JBoss module bootup process that it will trigger the static block initialization of that class in the context of the org.jboss.module module's classloader.
Thanks for the try, but this still doesn't work. I tried making a module for my jar and add it to org.jboss.modules as you suggested, but it still does not appear to be loaded.