I've updated the documentation to describe how to use custom authentication providers. Let us know if it's not clear.
Description relates to usage of modeshape in java se - json configuration files. What about modeshape on AS7?
It's still possible to do. I'll update the doc, but have a look at the XSD for the ModeShape subsystem in AS7, especially the "repository" element can contain a "authenticators" element with multiple "authenticator" elements inside, each of which contain a classname, module name, and logical name. Note again that order is important, and that the "module" attribute should contain the AS7 module name that contains your code.
Additional question - JCR spec says that
the content repository allows for authorization and authentication to be handled by JAAS (or another external mechanism) if the implementer so chooses.
So plain JAAS is not supported ony "external mechanism" described in https://docs.jboss.org/author/display/MODE/Custom+authentication+providers ?
JAAS is supported out-of-the-box, to configure it you just need to add a provider like:
, //this is predefined
, //this should be the name of your JAAS configuration
You can also see the implementation here: https://github.com/ModeShape/modeshape/blob/master/modeshape-jcr/src/main/java/org/modeshape/jcr/security/JaasProvider.java