The difference between the two is that for a security domain you present a username and a credential to the domain for verification during the authentication process - this means that this information needs to be available at the time an authentication check is going to be performed. 

The security realms on the other hand are implemented as CallbackHandlers handling various Callbacks, this means that it is no longer a case of present everything we know and get an authentication decision instead we can now integrate the authentication process at the transport level much more closely with the user stores without those stores needing to be aware of the details of the transport.

One example is the difference between how BASIC and DIGEST authentication can be used in the two approaches.  For security domains you end up with some of the Digest process living in a JBoss Web Authenitcator and then some of the HTTP specific items for verification need to live in the login module, if you want to add a new login module and still use DIGEST authentication then your new login module needs to be aware of HTTP Digets authentication.  The realms on the other hand just respond to callbacks based on a supplied username and return either the users password or a hash of their password allowing the transport specific checks to be made without the access to the user repository now needing to know about HTTP Digest - in addition this same realm implementation is then compatible with Remoting with uses SASL for authentication - to achieve this in security domains would require login modules to now be aware of two different transports in use.

There is a description of the security subsystem configuration in the manual: https://docs.jboss.org/author/display/AS71/Security+subsystem+configuration

It gives an overview of the security configuration and describes some of the login modules (see link at the bottom). One of it is the database module. So you can use it by configuring a security domain and referencing it in the security realm.

Regarding your security realm and domain question. I am not sure if it is really correct (it works at least) but here is what we are doing:

- We configured our own security domain containing a chain of login modules and mappings. This security domain is used by our ejbs.

- Additionally we configured our own security realm which references our security domain. The security realm is used to secure remoting.

This works in our current setup with only one application deployed on th AS. But I don't know how to configure the security realm and/or JBoss remoting if there are multiple applications using different security domains.