I have implemented my own CallbackHandler and in jboss-service.xml have the following:

<!-- JAAS security manager and realm mapping -->
 <mbean code="org.jboss.security.plugins.JaasSecurityManagerService"
 name="jboss.security:service=JaasSecurityManager">
 <attribute name="ServerMode">true</attribute>
 <attribute name="SecurityManagerClassName">org.jboss.security.plugins.JaasSecurityManager</attribute>
 <attribute name="DefaultUnauthenticatedPrincipal">anonymous</attribute>
 <attribute name="CallbackHandlerClassName">
 com.security.auth.module.DbCallbackHandler
 </attribute>
 <attribute name="DefaultCacheTimeout">1800</attribute>
 <attribute name="DefaultCacheResolution">60</attribute>
 <attribute name="DeepCopySubjectMode">false</attribute>
 </mbean>

When I start the server I get the following:

[ServiceConfigurator] Problem configuring service jboss.security:service=JaasSecurityManager
org.jboss.deployment.DeploymentException: Exception setting attribute javax.management.Attribute@869113 on mbean jboss.security:service=JaasSecurityManager; - nested throwable: (java.
ng.ClassNotFoundException: No ClassLoaders found for: com.security.auth.module.DbCallbackHandler)

The class file for DbCallbackHandler does exist in one of my deployed project jars. Does the implemented CallbackHandler class need to exist in some other JBoss directory?