ModeShape uses Externalizers and has code in the Schematic class to configure Infinispan to use specific externalizer implementations. It's not clear why this behavior is any different in the two environments, because those classes are all part of the ModeShape libraries.
ModeShape does have an Environment interface, and two different implementations: LocalEnvironment is used in JavaSE and RepositoryService that is part of ModeShape's subsystem for JBoss AS/EAP/Wildfly (depending upon version). One of the methods is to obtain the ClassLoader used for our components, but I don't know how this would relate to Infinispan not being able to find the correct marshallers.
I do think there is a way to configure Infinispan to use a specific class resolver, but I don't know what's the best way to use that in JavaSE. This might be a good question for the Infinispan forums.