I was able to capture traffic on the loopback interface using RawCap and I can see that there is a connection to port 8080 (GET /) that gets a HTTP 101 Switching Protocols response. There is a bunch of traffic after that but I can't tell what's going on. If anyone would find that useful I can upload it somewhere (doesn't look like I can attach it).
So at least it is making some kind of a connection.
I enabled TRACE logging in WildFly (just changed the main log level at the top of logging.properties), but it logs nothing while the client is trying to connect.
Well, I found the problem. My company provided PC had Trend Micro installed that includes some kind of Proxy (TmProxy) running on port 6999 that gets a copy of all traffic. That was somehow interfering with the remoting (my network captures included this in addition to the calls to 8080) - it was more obvious once I had a capture from a machine that didn't have a problem.
Disabled the services, rebooted and it works now.
Hopefully my day and a half of pain will be of use to someone else in future!