Did you see the warning near the top of the link that you provided? It says that TLS v1.2 isn't supported on the Oracle JDK in FIPS mode.
I did not actually see that. I only just found out we needed to use TLSv1.2. Do you know if there is anything other than NSS that could be used that wouldn't have this issue?
Or am I just screwed?
Have you researched SunJSSE that is provided with Java 8? It supports FIPS and TLS 1.2. I'm not aware of any detailed documentation on configuring it with WildFly, but a lot of the documentation in the page you linked to would also apply here. Obviously, pointing to the NSS configuration file would need to change, but configuring the different subsystems within Wildfly would not.
You may also need to install JCE (Java Cryptography Extension) which can be downloaded directly from Oracle. Export restrictions apply when using this library!
I tried adding the SunPKCS11-NSS to to that then moving the provider I added based on the original documentation to the bottom of the file as that looks like its still needed.
It also looks like I also need to add in the class for the provider and that it is part of the nss?
But I don't know what the class name would be.
So currently I'm looking for some "configuration for dummies" to try to make any kind of forward progress.
Upon further review, I don't see how this is any different really than my original configuration. Because I think all that is needed are the two lines one of which was the SunPKCS11 provider. And based on the link you provided I don't see where it states that TLSv1.2 would work. Are we not still using the sun provider's algorithms?
In the Mozilla NSS configuration, you added a provider, and modified an existing provider to the java.security file. The one that you added (that references the NSS configuration file) can be removed entirely. The other entry, that you modified, needs to be changed again. The directions for NSS say to change the internal SSL provider with the following parameter:
For using JSSE, you need to change that to simply "SunPKCS11-NSS".
SunPKCS11 is the provider class, and "NSS" specifies an instance name for the class. Take a look at JDK 8 PKCS#11 Reference Guide for more details on configuring the provider. Sections 2.2 and 2.3, in particular, seem to provide good information.
Edit: It looks like that article just configures JSSE to utilize Mozilla NSS. I don't know whether the strong SunJCE can support what you need, but it may be worth researching that. If it does, then the JSSE configuration would need to point to SunJCE.
One lat suggestion. There's a cyptographic library called Bouncy Castle that is FIPS 140-certified. You can try to use that as your SSL provider, and place the needed BC jars into the classpath. The website is: http://www.bouncycastle.org
There's a user's guide at: https://www.bouncycastle.org/fips/BCUserGuide.pdf
Section 2.2 (the bottom) describes how to configure the java.security file, including the classname of the provider.