According to the release notes of Java 8 U121 JRE:
Serialization Filter Configuration
Serialization Filtering introduces a new mechanism which allows incoming streams of object-serialization data to be filtered in order to improve both security and robustness. Every ObjectInputStream applies a filter, if configured, to the stream contents during deserialization. Filters are set using either a system property or a configured security property. The value of the "jdk.serialFilter" patterns are described in JEP 290 Serialization Filtering and in <JRE>/lib/security/java.security. Filter actions are logged to the 'java.io.serialization' logger, if enabled.
You can set this property with Wildfly 11 using the security-properties in the Elytron config. Reference: How can I set a property on java.security?
I can also set the "jdk.serialFilter" property directly in <JRE>/lib/security/java.security, and it is working fine. The serial filter is used, e.g. during startup of Wildfly.
But the filter is never called during unmarshalling of remote EJB invocations. It appears to me that jboss-marshalling is doing deserialization on its own, thus circumventing most of the deserialization stuff that is provided by the JRE, including the filter.
So, the question is whether there is either some way to use the deserialization filter provided by the JRE, or some other means to incorporate whitelisting/blacklisting during unmarshalling of remote EJB calls.
jboss-marshalling is using RiverUnmarshaller found here: jboss-marshalling/river/src/main/java/org/jboss/marshalling/river at 643781486fbd30ef8c8b4833e3bda72f681074f8 · jboss-re… It looks to be just a reimplementation of the java.io.ObjectOutputStream class for some reason.
I'm not sure, maybe a Red Hat dev can talk to more of on the why. My guess is that they did this before JEP 290 made its way in to the SDK.
You might be able to submit a feature request for Wildfly for this.