Well solaris JVM & NIO are notoriously buggy, so first step would be to upgrade JDK to latest (u131 atm)
and than we can look in other options.
Also upgrading WildFly to 10.1 would not hurt
Besides the upgrade, do you have any recommendations on how to tackle/investigate this? The issue appeared over night, without any modifications on the application itself.
You may try increasing direct buffer memory using -XX:MaxDirectMemorySize=512m JVM option. However, consider to upgrade to JDK to latest update and Wildfly too.