wildfly10 uses undertow as the default web container inplace of web subsystem. In wildfly 10, you now configure thread pools for the undertow subsystem by referencing a worker that is defined in the io subsystem. You may refer link Undertow subsystem configuration - WildFly 10 - Project Documentation Editor for more information.
undertow is done using non blocking io (NIO2) where jbossweb was using blocking io (BIO)
as such there are many configuration options that don't apply anymore, one of them is reject count among others.
you can stil tune thread pools, they are configured in IO subsystem under workers.
but keep in mind that with NIO thread pools have different function than with BIO.
if you noting else you have two thread pools, "IO thread pool" and "worker (task) thread pool"