This has to do with https://jira.jboss.org/jira/browse/JBAS-5815.
In short, when configuring a JRMPInvoker with a RMISSLServerSocketFactory [1] in conf/jboss-service.xml, a NPE is seen due to a failure to initialize the security domain.
The securityDomain attribute of RMISSLServerSocketFactory is set by the SecurityDomainEditor, that is responsible for converting the java:/jaas/domain String to the actual SecurityDomain instance that must be used by the DomainServerSocketFactory. This editor is installed by the JaasSecurityManagerService MBean that is specified in conf/jboss-service.xml.
When a JRMPInvoker is also configured in this file, by the time the RMISSLServerSocketFactory is constructed the SecurityDomainEditor is not available because JaasSecurityManagerService has been created but not yet started. As a result, a null security domain is set and this causes an NPE when a ServerSocket is created later on.
A workaround is to create a sar with the customized invoker or configure it somewhere else, like deploy/jrpm-invoker-service.xml or remoting-service.xm (AS 5). However we've been getting reports from many people complaining about this failure. So either we include in the documentation that customized JRMP invokers with SSL should not be configured in conf/jboss-service.xml or we find a fix - a way to install SecurityDomainEditor before the RMISSLServerSocketFactory is created. Any thoughts?
PS: In AS5 we should probably try to configure the JRMPInvoker as a MC bean instead of an MBean and then inject the security domain directly when SSL is needed. This would eliminate the need for the SecurityDomainEditor, not to mention it would be a much cleaner way of configuring things as no JNDI path (java:/jaas/domain) would have to be exposed.
[1] - http://wiki.jboss.org/wiki/JRMPInvoker
