Version 1

    Problem: Intermittent Invalid User Errors on JBoss 6 Using the LdapExtLoginModule and Active Directory

     

    We had a problem sililar to this post...

     

    Hello everyone!

     

    Hope somebody can help me:

     

    Since some time I have the issue that I receive sometimes (and that is the mostly strange thing) the following exception:

     

    javax.ejb.EJBAccessException: Invalid User

        at

    org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:161)

        at

    org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

        at

    org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:41)

     

     

    What is strange: The application runs for several days, even several weeks .. and suddenly the error occurs. A JBoss-restart solves the issue and everything is fine again.

     

    Maybe it has something to do with security-changes we made (as the error just occured some time after these changes) regarding this description https://community.jboss.org/wiki/SecureTheJmxConsole:

    What we did: We modified the following files to make sure that for jmx-console and web-console login a password is forced:

    /opt/jboss/jboss-6.0.0.Final/common/deploy/jbossws-console.war/WEB-INF/jboss-web.xml

    /opt/jboss/jboss-6.0.0.Final/common/deploy/jbossws-console.war/WEB-INF/web.xml

    /opt/jboss/jboss-6.0.0.Final/common/deploy/jmx-console.war/WEB-INF/jboss-web.xml

    /opt/jboss/jboss-6.0.0.Final/common/deploy/jmx-console.war/WEB-INF/web.xml

    /opt/jboss/jboss-6.0.0.Final/server/default/conf/props/jmx-console-users.properties

    /opt/jboss/jboss-6.0.0.Final/server/default/conf/props/jmx-console-roles.properties

     

    Again .. the strange thing is that everything works well for a several, not defined periode of time. And after a restart everthing is fine again (and we also can't reproduce the error but we can just wait until this happens again).

     

    Hope somebody has some ideas.

     

    Thanks!

     

    KR,

    James

     

    How we approached troubleshooting and fixing this isuue:

     

    The basic idea is this...  JBoss is open source so we can get the source for the LdapExtLoginModule.  JBoss also allows us to define and install our own custom login module. This is documented and not too difficult. Combining these two ideas, one can get the JBoss code, add some logging and debugging information and install it in the  application server.  Which is what I did.

     

    Once I had that working, I also recognized that JBoss has a cache of the LDAP lookups.  So if you want to exercise your LDAP you must shut caching off.  To shut caching off

     

    find the file server\default\deploy\legacy-conf-service.xml

     

    find the section for

        <mbean code="org.jboss.security.plugins.JaasSecurityManagerService" name="jboss.security:service=JaasSecurityManager">

     

    the set the following two values to zero

     

     

    <attribute name="DefaultCacheTimeout">0</attribute>


    <attribute name="DefaultCacheResolution">0</attribute>

     

    Write a test case that will exercise your login.  In my case I have a secured EJB.  So I wrote a client application that called a method on the EJB in a loop with a 1 second pause between executions.  Now I am authenticating every call against my LDAP.

     

    Now I can really see it fail.  I start getting about one failure every 2 minutes.

     

    At this point, I have enough logging to see that my root problem is a socket timeout on my LDAP -- Active Directory.

     

    It appeard to us that we had one bug in the finally block of this method.

    rolesSearch(InitialLdapContext ctx, SearchControls constraints, String user, String userDN, int recursionMax, int nesting)  throws NamingException

     

    First the code makes this call

     

    NamingEnumeration results = ctx.search(rolesCtxDN, roleFilter, filterArgs, constraints);

     

    In the finally block is has

    if( results != null){  results.close(); }

     

    The results.close method is where the connection exception is being thrown.  So, what I found is that the LDAP was authenticating, fetching all the roles, and we were encountering an exception after successfully doing all the difficult and important work.

     

    The documentation for the close method shows this:

     

    Closes this enumeration.  After this method has been invoked on this enumeration, the enumeration becomes invalid and subsequent invocation of any of its methods will yield undefined results. This method is intended for aborting an enumeration to free up resources. If an enumeration proceeds to the end--that is, until hasMoreElements() or hasMore() returns false-- resources will be freed up automatically and there is no need to explicitly call close().

     

    So, it seems reasonable to change the code to this:

     

    if( results != null && results.hasMore())

                    try{

                        results.close();

                    } catch (Exception e){

                        log.error("Error closing naming enumeration", e);

                    }

             }

     

    This seems to resolve one problem.

     

    Next, we see that there is a significant performance problem in the method

    protected void rolesSearch(InitialLdapContext ctx, SearchControls constraints,

                    String user, String userDN, int recursionMax, int nesting)

                    throws NamingException

     

    To determine this, we put Microsoft Network Monitor between the JBoss Server and the LDAP and found that if the role is a DN   -- login-config.xml file has <module-option name="roleAttributeIsDN">true</module-option>   --   then for a user with n roles, this method makes at least n + 1 network round trips.  Essentially the program gets a list of roles from the LDAP but a role is in a form that shows the entire DN (Distinguished Name).  The DN includes the part we are interested in the CN, which is the role.  Rather than parsing the role portion out of the DN in code, the login module makes a call to the LDAP for each role simply to ask the LDAP to return the CN portion the from the DN.  So if I have n roles this is n socket connections to the LDAP.  In our case, this turned out to be one of the most frequent causes of failure, as it is not uncommon for a user to have in excess of 30 roles as our LDAP is configured such that it returns all roles for a user, not just the ones that pertain to our Java EE applications.  With this much load, our LDAP would either not accept the connection, or would simply time out.  Accordingly we would get an Invalid User error.  The solution that seems to be working is to refactor this code to parse the CN out of the DN, thus eliminating a large number of calls to the LDAP.  This took about 10 minutes to code.  So it's not a hard problem to solve.  After this change our network round trips drop, we authenticate much more quickly, and we seem to stop having the Invalid User problems.  Now, when I look at Network Monitor I see that the network calls are down to bind using bind account, bind using the user's account (validates user and password), then one call to fetch the list of user roles.

     

    I would note, for a general purpose solution the JBoss approach may make sense, as it's a robust approach that should generally work for any organization, whereas my approach is tailored to my organization.  However, the reality is that the out of the box solution is quite inefficient.

     

    I would also add that there is another apparent probelm in the LdapExtLoginModule:

     

    When any socket timeout or socket connection exception is thrown, it prevents the validatePassword method from returning normally one of {true, false}, which in turn prevents  a second backup provider from being able to fulfill the request.  In my testing I found that trying to provide a second LDAP or file based backup provider failed when a socket connection exception was encountered.

     

    So, if you are having this problem, use the LdapExtLoginModule as a starting point for your own custom provider.  Customize the provider to your needs.  The JBoss provided LdapExtLoginModule is perhaps over generalized for your needs and can be improved.

     

    Good Luck,

    Steve

     

     

    This article was generated from the following discussion: JBoss 6 - Invalid User-issue