You need to modify deploy/ejb3-connectors-jboss-beans.xml and change the invokerLocator to use a sslsocket instead of a plain socket. Here is an example configuration:

<?xml version="1.0" encoding="UTF-8"?>
<!--

  EJB3 Connectors

-->

<deployment xmlns="urn:jboss:bean-deployer:2.0">
    <!--  We don't want the AOPDependencyBuilder  -->
    <annotation>@org.jboss.aop.microcontainer.annotations.DisableAOP</annotation>

    <bean name="JaasSecurityDomain:ejb3" class="org.jboss.security.plugins.JaasSecurityDomain">
      <constructor>
         <parameter>ejb3</parameter>
      </constructor>
      <property name="keyStoreURL">localhost.keystore</property>
      <property name="keyStorePass">changeit</property>
      <property name="keyStoreAlias">ejb3</property>
      <property name="trustStoreURL">client.truststore</property>
      <property name="trustStorePass">changeit</property>
      <!-- introduce a JMX annotation to export this bean as an MBean -->
      <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.security:service=JaasSecurityDomain,domain=ejb3",exposedInterface=org.jboss.security.plugins.JaasSecurityDomainMBean.class)</annotation>
   </bean>
  <!--

    JBoss Remoting Connector

    Note: Bean Name "org.jboss.ejb3.RemotingConnector" is used
    as a lookup value; alter only after checking java references
    to this key.

  -->
  <bean name="org.jboss.ejb3.RemotingConnector"
    class="org.jboss.remoting.transport.Connector">

    <property name="invokerLocator">

      <value-factory bean="ServiceBindingManager"
        method="getStringBinding">
        <parameter>
          jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3
        </parameter>
        <parameter>
          <null />
        </parameter>
        <parameter>sslsocket://${jboss.bind.address}:${port}?timeout=300000</parameter>
        <parameter>
          <null />
        </parameter>
        <parameter>3873</parameter>
      </value-factory>

    </property>
    <property name="serverConfiguration">
      <inject bean="ServerConfiguration" />
    </property>
    <property name="serverSocketFactory">
      <inject bean="SSLServerSocketFactory" />
    </property>
  </bean>

  <bean name="SSLServerSocketFactory" class="org.jboss.security.ssl.JaasSecurityDomainServerSocketFactory">
    <property name="needsClientAuth">true</property>
  </bean>

  <!-- Remoting Server Configuration -->
  <bean name="ServerConfiguration"
    class="org.jboss.remoting.ServerConfiguration">
    <property name="invocationHandlers">
      <map keyClass="java.lang.String" valueClass="java.lang.String">
        <entry>
          <key>AOP</key>
          <value>
            org.jboss.aspects.remoting.AOPRemotingInvocationHandler
          </value>
        </entry>
      </map>
    </property>
  </bean>

</deployment>

Notice that to use JaasSecurityDomainServerSocketFactory I needed to create the factory in a bean (SSLServerSocketFactory) and inject it in the serverSocketFactory property of the connector. We need to set the needsClientAuth property to true to require client authentication. The name of the security domain used by this socket factory is set up by the org.jboss.security.ssl.server.domain.name system property, so I had to add this to $JBOSS_HOME/bin/run.conf:

JAVA_OPTS="$JAVA_OPTS -Dorg.jboss.security.ssl.server.domain.name=ejb3"

The configuration for this security domain is also included in a bean (JaasSecurityDomain:ejb3).