Random "Security Context is null" after using SecurityClient

[snacker]

We have converted most of our ejb's from 2.1 to 3.0.

In order to get the client security to work we had to use the following code to setup the client's context for the ejb3:

SecurityClient client = SecurityClientFactory.getSecurityClient();
                client.setSimple( username, password );
                client.setVmwideAssociation( true );
                client.login();

Once that code has been executed and the ejb invoked we start seeing random errors such as this:

15:01:01,035 ERROR [CoyoteAdapter] An exception or error occurred in the container during the request processing
java.lang.IllegalStateException: Security Context is null
        at org.jboss.web.tomcat.security.SecurityAssociationActions$PopRunAsRoleAction.run(SecurityAssociationActions.java:156)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.jboss.web.tomcat.security.SecurityAssociationActions.popRunAsIdentity(SecurityAssociationActions.java:302)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:193)
        at org.jboss.web.tomcat.service.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:135)
        at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:94)
        at org.jboss.web.tomcat.service.session.LockingValve.invoke(LockingValve.java:62)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
        at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
        at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:619)

The error is "random"... and it occurs on any of the servlets that we have configured.

For example we've had 32,000 requests (according to http://myserver:8090/status?full) but only 99 of these errors.

It happens for the servlets/jsps whether they are called directly from the server or via apache by way of mod_jk (ajp3).

Does anyone know why this would happen?

Do we have to setup some "default" security for the servlets/jsps?

[snacker]

Does anyone know why this might be happening?

I contacted jboss sales at jboss.com to see about getting a support contract, but still haven't gotten a response yet.

[jaikiran]

Can't really say what's wrong. You might want to check the logs to see if that provides any clues.

[snacker]

There is not much else in the jboss log that look related to this.

We have also seen intermittent errors such as:

14:08:29,259 ERROR [ServerInvoker] Error executing server oneway invocation request: InvocationRequest[61db622f, JMS, org.jboss.jms.wireformat.SessionSendRequest@a0b2dda]
java.lang.IllegalArgumentException: Subject passed is null
    at org.jboss.security.plugins.JBossAuthorizationManager.getCurrentRoles(JBossAuthorizationManager.java:384)
    at org.jboss.security.plugins.JBossAuthorizationManager.getCurrentRoles(JBossAuthorizationManager.java:378)
    at org.jboss.security.plugins.JBossAuthorizationManager.doesUserHaveRole(JBossAuthorizationManager.java:184)
    at org.jboss.jms.server.jbosssx.JBossASSecurityMetadataStore$1.run(JBossASSecurityMetadataStore.java:251)
    at org.jboss.jms.server.jbosssx.JBossASSecurityMetadataStore$1.run(JBossASSecurityMetadataStore.java:248)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.jboss.jms.server.jbosssx.JBossASSecurityMetadataStore.authorize(JBossASSecurityMetadataStore.java:247)
    at sun.reflect.GeneratedMethodAccessor486.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
    at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
    at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
    at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
    at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
    at javax.management.StandardMBean.invoke(StandardMBean.java:391)
    at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:164)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
    at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
    at $Proxy365.authorize(Unknown Source)
    at org.jboss.jms.server.container.SecurityAspect.check(SecurityAspect.java:304)
    at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:155)
    at sun.reflect.GeneratedMethodAccessor487.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
    at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
    at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
    at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897)
    at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:2075)
    at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
    at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:2086)
    at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:867)
    at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:106)
    at org.jboss.remoting.Client.invoke(Client.java:1927)
    at org.jboss.remoting.Client.invoke(Client.java:770)
    at org.jboss.remoting.Client.invokeOneway(Client.java:819)
    at org.jboss.remoting.Client.invokeOneway(Client.java:1005)
    at org.jboss.remoting.Client.invokeOneway(Client.java:996)
    at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180)
    at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165)
    at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:495)
    at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeTarget(ClientSessionDelegate$send_6145266547759487588.java)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
    at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661)
    at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleSend_2080159813.invoke(SessionAspect_z_handleSend_2080159813.java)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
    at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
    at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
    at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:269)
    at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect_z_handleSend_2080159813.invoke(ProducerAspect_z_handleSend_2080159813.java)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
    at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
    at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
    at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
    at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
    at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
    at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
    at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
    ...

I seems related, because it is in the "security" package, but again... why would it be ok for 1000's of calls then get a few of these errors?

[snacker]

Does anyone know where the source code for org.jboss.security.SecurityContextAssociation is?

The call to getSecurityContext() is "randomly" returning null.

org.jboss.web.tomcat.security.SecurityAssociationActions

...

static SecurityContext getSecurityContext()
   {
      //GetSecurityContextAction action = new GetSecurityContextAction(securityDomain);
      return (SecurityContext)AccessController.doPrivileged(new PrivilegedAction()
      {
         public Object run()
         {
            return SecurityContextAssociation.getSecurityContext();
         }
       });
   }

...

private static class PopRunAsRoleAction implements PrivilegedAction
   {
      static PrivilegedAction ACTION = new PopRunAsRoleAction();
      public Object run()
      {
         //RunAsIdentity principal = SecurityAssociation.popRunAsIdentity();
         SecurityContext sc = getSecurityContext();
         if(sc == null)
            throw new IllegalStateException("Security Context is null");
         RunAs principal = null;
         principal = sc.getOutgoingRunAs();
         sc.setOutgoingRunAs(null);
         return principal;
      }
   }

[peterj]

In the source download for AS 5.1.0.GA, you can find the source for that class in jboss-5.1.0.GA-src/thirdparty/jboss/jboss-security-spi/lib/jboss-security-spi-sources.jar

[snacker]

Thanks!

So... here is the code for the getSecurityContext() method:

org.jboss.security.SecurityContextAssociation

...

public static SecurityContext getSecurityContext()
   {
      SecurityManager sm = System.getSecurityManager();
      if(sm != null)
         sm.checkPermission(GetSecurityContextPermission);
     
      if(!SERVER)
         return securityContext;
     
      return securityContextLocal.get();
   }

...

Here is a beanshell script (sec.bsh) to test these calls:

sm = System.getSecurityManager() ;
System.out.println(" XXX: system security manager : " + ( null == sm ? "null" : ( sm + " / " + sm.getClass() ) ) );
System.out.println(" XXX: is client               : " + org.jboss.security.SecurityContextAssociation.isClient() );

sm = org.jboss.security.SecurityContextAssociation.getSecurityContext();
System.out.println(" XXX: jboss  security manager : " + ( null == sm ? "null" : ( sm + " / " + sm.getClass() ) ) );

here are the results:

15:09:10,818 INFO  [STDOUT]  XXX: system security manager : null
15:09:10,955 INFO  [STDOUT]  XXX: is client               : false
15:09:10,962 INFO  [STDOUT]  XXX: jboss  security manager : null

So securityContextLocal.get() should be returning an "InheritableThreadLocal<SecurityContext>" instance.

I'm going to check the tomcat code to see where it is setting a security context.

[snacker]

This "Security Context is null" looks "impossible" from going through the code.

From what I can tell org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(Request request, Response response, HttpEvent event) makes sure there is a SecurityContext set before it calls the "invoke(...)" method.

It should be setting the SecurityContext in an InheritableThreadLocal variable in org.jboss.security.SecurityContextAssociation.

So... I'm not sure how this is randomly getting cleared.

Is there possibly a thread which might try to clean up ThreadLocal data or something?

That's the only way I can think of that could cause this.

I have changed SecurityContextEstablishmentValve.process(Request request, Response response, HttpEvent event) to retry the request up to 4 times if the "Security Context is null" error is thrown and to log a warning for now.

[snacker]

With the changes that I've made I'm seeing the following in the logs when one of these errors occurs:

1. sc = "org.jboss.security.plugins.JBossSecurityContext()CLIENT)"
2. createdSecurityContext = true
3. cachedContext = "org.jboss.security.plugins.JBossSecurityContext()CLIENT)"
4. SecurityAssociationActions.getSecurityContext() = null

OR

1. sc = "org.jboss.security.plugins.JBossSecurityContext()jboss-web-policy)"
2. createdSecurityContext = true
3. cachedContext = null
4. SecurityAssociationActions.getSecurityContext() = null

So from the above it "should" have a security context, except (#4 above) something has nulled the ThreadLocal that was set in SecurityAssociationActions.getSecurityContext() just a few calls earlier.

The 2nd time it runs it does not get the "Security Context is null" error.

(again... this error only happens randomly; most of the time it does not get these errors)

I am not sure why one context uses "CLIENT" as the domain while another uses "jboss-web-policy" though.

I have attached the "original" SecurityContextEstablishmentValve.java file.

[subramaniam.venkat]

Hell Nathan,

I am also facing the same problem. In my application, when the doesUserHaveRoles is invoked on the JAASSecurityManagerMBean the first time, the authentication suceed. The second time when requested for authentication for the same Pricinpal, the SecurityAssociation.getCredential() returns "null".

When I enabled the org.jboss.security on the logging I found the below stacktrace.

https://community.jboss.org/thread/154809

When I put break points and debug in eclipse, for me too SecurityAssociation.getCredential() returns a "null" credential and therefore, the authentication fails.

If you get any help, please post the same on this thread.

Hello All,

Please do have a look the below thread too.. Please provide your comments.

https://community.jboss.org/thread/154809

Thanks & Warm Regards,

Anand.

[snacker]

I will post whatever I can find out about this issue.

Hopefully we can get this resolved sooner than later.

[snacker]

I changed the code to verify that the security context exists directly after it creates it.

That is before it even calls the "invoke(...)" method of the next object in the pipeline.

This code is shown below in the for loop interating over 'x' below:

   private void process(Request request, Response response, HttpEvent event)
   throws IOException, ServletException
   {
      outer_for:
         for( int i=0; i<4; ++ i ){

            SecurityContext cachedContext = null;
           
            boolean createdSecurityContext = false;
            //Set the security context if one is unavailable
            SecurityContext sc = SecurityAssociationActions.getSecurityContext();
            if(sc != null &&
                  sc.getSecurityDomain().equals(configuredSecurityDomainName) == false)
            {
               cachedContext = sc;
               SecurityContext newSC = createSecurityContext();
               SecurityAssociationActions.setSecurityContext(newSC);
               createdSecurityContext = true;
            }
           
            if(sc == null)
            {
               sc = createSecurityContext();
               SecurityAssociationActions.setSecurityContext(sc);
               createdSecurityContext = true;
            }
           
            try
            {
           
               for( int x=0; x<5; ++ x ){
                  SecurityContext sc2 = SecurityAssociationActions.getSecurityContext();
                  if( null == sc2 ){
                     System.out.println( "sc2 is null : i=" + i + " : x=" + x );
                     try{ Thread.sleep( 100 ); }catch( Exception ignore ){}
                     continue outer_for;
                  }
               }
              
               // Perform the request
               if(event == null)
                  getNext().invoke(request, response);
               else
                  getNext().event(request, response, event);
              
               return;
            }

            catch( IllegalStateException e )

            {
                if( ...is a "Security Context is null" error... ){

                    continue;

                }

                throw e;

            }

            finally
            {
               SecurityRolesAssociation.setSecurityRoles(null);
               if(createdSecurityContext)
               {
                  SecurityAssociationActions.clearSecurityContext();
               }
               if(cachedContext != null)
                  SecurityAssociationActions.setSecurityContext(cachedContext);
            }
         }
   }

What I'm seeing are messages such as (not in this order... they appear randomly):

    sc2 is null : i=0 : x=0

    sc2 is null : i=0 : x=1

    sc2 is null : i=0 : x=2

    sc2 is null : i=0 : x=3

    sc2 is null : i=0 : x=4

    sc2 is null : i=1 : x=0

    sc2 is null : i=1 : x=1

    sc2 is null : i=1 : x=2

    sc2 is null : i=1 : x=3

    sc2 is null : i=1 : x=4

So if x=1,2,3,4 that means that SecurityAssociationActions.getSecurityContext() has returned a non-null value at least once after it was set previously, but something has been changed so it now returns a null value.

I haven't yet seen where i is greater than 1 yet... so at least it is being handled by the retry at some point.

However, I would really appreciate it if someone could give us some insite as to why this would be happening.

It could be we have something configured incorrectlly... any help would be appreciated.

[snacker]

It looks like org.jboss.security.SecurityContextAssociation.setClient() is being called somewhere.

The call to this method sets the static SERVER member to false.

Once it has been set to false there are no methods to unset it.

If SERVER is false, then the SecurityContextAssociation uses a static variable to hold the security context.

This variable does NOT use a ThreadLocal.

Since multiple threads are setting and clearing this, I would expect the threads to clobber each others sessions.

Does anyone know what would be causing SecurityContextAssociation.setClient() to be called?

Is there a config setting we might be missing?

It looks like org.jboss.security.SecurityAssociation could cause this if there was a race condition while initializing, but...?

[snacker]

I think we have figured out what is going on now.

The code for doing the client login is put in a utility method which takes the username and password:

public static void login( String username, String password )throws Exception{

SecurityClient client = SecurityClientFactory.getSecurityClient();
client.setSimple( username, password );
client.setVmwideAssociation( true );
client.login();

}

This works fine, but the problem is the call to "client.setVmwideAssociation( true );".

One of our many EJB implementations calls this login on the jboss side in order to do execute a method which requires the login to be there.

Calling setVmwideAssociation( true ) in turn calls SecurityContextAssociation.setClient() which should not be done within the jboss server, and that is what is causing all of our grief.

So this is our fix:

private static final boolean s_isVmWideLoginEnabled = Boolean.getBoolean( "isVmWideLoginEnabled" );

...

public static void login( String username, String password )throws Exception{

SecurityClient client = SecurityClientFactory.getSecurityClient();
client.setSimple( username, password );

if( s_isVmWideLoginEnabled ){
    client.setVmwideAssociation( true );

}
client.login();

}

Then we set -DisVmWideLoginEnabled=true on the command line only for the client-side apps.

[anil.saldhana]

I am sorry Nathan.  I just saw this thread late.  First thing I would have told you is to remove the vmwide association.  That is a dangerous thing. We want thread based security context to be propagated in a server environment.  If you specify vm wide association, then there may be some unseen security checks passing in components that are not configured properly.  The AS is a mightly jungle with all types of services.