JRMPInvoker has a method private void loadCustomSocketFactories()

If you defined a custom serverSocketFactory, an instance of this factory is made
and the BindAddress is set:
if( serverSocketFactoryName != null )
{
Class ssfClass = loader.loadClass(serverSocketFactoryName);
serverSocketFactory = (RMIServerSocketFactory) ssfClass.newInstance();
if( serverAddress != null )
{
// See if the server socket supports setBindAddress(String)
try
{
Class[] parameterTypes = {String.class};
Method m = ssfClass.getMethod("setBindAddress", parameterTypes);
Object[] args = {serverAddress};
m.invoke(serverSocketFactory, args);
}
catch (NoSuchMethodException e)
{
log.warn("Socket factory does not support setBindAddress(String)");
// Go with default address
}
catch (Exception e)
{
log.warn("Failed to setBindAddress="+serverAddress+" on socket factory", e);
// Go with default address
}
}
}

So far, so good.
But if you use org.jboss.security.ssl.RMISSLServerSocketFactory
you also have to invoke the setSecurityDomain-method.

Otherwise the RMISSLServerSocketFactory is constructed WITHOUT
a securityDomain. As a result, a DomainServerSocketFactory is constructed:
domainFactory = new DomainServerSocketFactory();

Instead of using for instance:
domainFactory = DomainServerSocketFactory.getDefault();

So the securityDomain always remains null and you get a NullPointerException
upon startup of jBoss.

In other words: it is impossible to use the provided classes to have
RMI working over SSL.

A quick patch would be to use DomainServerSocketFactory.getDefault()
in class RMISSLServerSocketFactory.
A more elegant solution would be to check for a SecurityDomain in the
JRMPInvoker class and to set this variable. This way people could chose
there own domain instead of using java:/jaas/other.

I hope I have given you enough info to solve this bug,
Bruno