On the server side, we probably should be able to configure the security like our other subsystems (e.g. so we can share the same userid + password with the jmx console if desired or have a separate identity).

One reference that describes how jconsole passed the authentication is here: http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html :

"JConsole will pass the user name and password as client credentials as authentication to the RMI connector server by setting the "jmx.remote.credentials" property to these values in the environment map for establishing the connection."

Perhaps we can support passing credentials the JConsole way and the org.jboss.security.SecurityContext way.

To me the solution is simple. Just add a JMX authenticator that passes the jconsole username and password through to our Authenticator. We could support additional auth mechanisms, but we are limited to whatever RMI gives us. We need a r3 based impl for that.

Right. JConsole currently sends a two element String array.

Should we have a conf/jmx-users.properties or just use the existing jmx-console-users.properties?  Same question for jmx-roles.properties versus using the existing jmx-console-roles.properties.

Or should I be different and plug these properties directly into the jmx-jboss-beans.xml?

probably not jboss-beans.xml, need to plug into login-config.xml as well.

Yeah you don't need a properties file, just something like this pseudocode:

class YourAuth implements JMXAuthenicator {
    String configName = ...get this from jboss-beans.xml...;
    LoginContext context = ...get this from configName...;
    YourCallbackHandler cbh = ...has writable properties for user/pass...;

    [...]
    public synchronized Subject authenticate(Object creds) {
        String user, pass;
        user = ((String[])creds)[0]; pass = ((String[])creds)[1];
        // validate user/pass (not shown)
        // now put it on cbh
        cbh.setUserName(user); cbh.setPassword(pass);
        // authenticate:
        context.login();
        Subject s = context.getSubject();
        s.setReadOnly();
        return s;
    }

}

Do you know where the current code lives that performs the authentication for any of the conf/props/* ?

I can follow that example.

Amazing, I keep getting answers to the question before asking it. 

The only thing I want to chime in here is that JBossSecurityContext must not cross the wire. In fact it should not be serializable at all. It's more than just a security context and too heavy for remote usage.