Following steps would give an idea on basic steps involved in creating a custom login module(following example extends out of the box module 'UsernamePasswordLoginModule')

1)create a security domain in standalone.xml

eg:

                <security-domain name="customSecurity" cache-type="default">

                    <authentication>

                        <login-module code="com.CustomModule" flag="required"/>

                    </authentication>

                    <authorization>

                        <policy-module code="PermitAll" flag="required"/>

                    </authorization>

                </security-domain>

2)create custom login module

eg:

          public class CustomModule extends UsernamePasswordLoginModule{

                 @Override

                 protected Group[] getRoleSets() throws LoginException {

                      /**any role could be returned as the security domain declared in above standalone.xml allows permission for all roles*/

                      SimpleGroup group = new SimpleGroup("Roles");

                       try {

                           group.addMember(new SimplePrincipal("noGroup"));

                       } catch (Exception e) {

                           throw new LoginException("Failed to create group member for " + group);

                       }

                       return new Group[] { group };

                 }

                 @Override

                 protected boolean validatePassword(String inputPassword, String expectedPassword) {

                      /**do actual validation,'this.getUsernameAndPassword()' returns username,&password sent to this module*/

                      return true;

                   }

                 @Override

                 protected String getUsersPassword() throws LoginException {

                      /** 'this.getUsernameAndPassword()' would provide username,&password sent to this module*/

                      /**this method could be used if user entered password need to be hashed before validation is done by above 'validatePassword' method */

                      return "sri";

                 }

          }

3)ways to deploy Custom Login Module

     a)adding as a new module in wildfly(simple steps,plz check docs on how to add a new module in wildfly)

     or

     b)if WAR artifact uses this Login Module,package this as a jar within the artifact

4)if WAR artifact uses this Login Module,make it aware of this security domain through 'jboss-web.xml',and place this xml inside WEB-INF directory

      eg:

          <jboss-web>

              <security-domain>java:/jaas/customSecurity</security-domain>

          </jboss-web>

5)invoke custom login module from filter or servlet to perform authentication after clicking login button of any custom UI login screen

     eg:

          httpRequest.login("wildfly", "meowfly");

    note:need servlet 3 api libraries to use above login method

6)once validated by login module(ie validatePassword() returns true,& getRoleSets() matches with roles declared in standalone.xml),Principal object would be available to EJBs,interceptors MAGICALLY

     eg:

          @Resource

          private javax.ejb.SessionContext sessionContext;

          String caller = sessionContext.getCallerPrincipal().getName();

hope above steps give an idea on how to override other out of the box login modules