tipically the message "The data area passed to a system call is too small" is desceptive, it just means there's another underlying error.
Keep in mind that installing the service creates a key in the Window Registry (browseable with regedit) under
HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Apache Software Foundation\Procrun 2.0\<server id>
In particular under there there's a stop command like
/c <jboss home>\bin\jboss-cli.bat --controller=localhost:9990 --connect --command=:shutdown
The stop port (9990) is fixed but if you have changed it in your standalone.xml you
need to change the registry key as well by hand.
Also, keep in mind that jboss-cli.bat uses the system variable JAVA_HOME.
Make sure that the jdk under JAVA_HOME has a version compatible Wildfly's .class files.