ArrayIndexOutOfBoundsException in SecurityAssociation

[milx]

Hello
I just upgraded from 3.2.7 to 4.0.3, and I see that the org.jboss.SecurityAssociation class is throwing java.lang.ArrayIndexOutOfBoundsException for no particular reason I can think of.

The error occurs when I create new home interfaces for my stateless EJBs, but it's not consistent, as most times the EJBs are created successfully. The error occurs with multiple different EJBs, and causes an insufficient method permissions error with principalRoles=null to be reported back to the client. I took a look at the source, and I see that the exception occurs in the pop() method of the inner class RunAsThreadLocalStack in SecurityAssociation:

 RunAsIdentity pop()
 {
 ArrayList stack = (ArrayList) local.get();
 RunAsIdentity runAs = null;
 int lastIndex = stack.size() - 1;
 // This is where the exception occurs
 if (lastIndex >= 0)
 runAs = (RunAsIdentity) stack.remove(lastIndex);
 return runAs;
 }

Could the cause be unsynchronized access to the ArrayList stack? It looks to me like the size of the List has changed between stack.size() and stack.remove(). Here is an excerpt of the stacktrace:

java.lang.IndexOutOfBoundsException: Index: 72, Size: 72
 at java.util.ArrayList.RangeCheck(ArrayList.java:547)
 at java.util.ArrayList.remove(ArrayList.java:390)
 at org.jboss.security.SecurityAssociation$RunAsThreadLocalStack.pop(SecurityAssociation.java:623)
 at org.jboss.security.SecurityAssociation.popRunAsIdentity(SecurityAssociation.java:544)
 at org.jboss.ejb.plugins.SecurityActions$10.pop(SecurityActions.java:156)
 at org.jboss.ejb.plugins.SecurityActions.popRunAsIdentity(SecurityActions.java:313)
 at org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:135)
 at org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:121)
 at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(ProxyFactoryFinderInterceptor.java:93)
 at org.jboss.ejb.SessionContainer.internalInvokeHome(SessionContainer.java:613)
 at org.jboss.ejb.Container.invoke(Container.java:894)
 at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invokeHome(BaseLocalProxyFactory.java:344)
 at org.jboss.ejb.plugins.local.LocalHomeProxy.invoke(LocalHomeProxy.java:118)
 at $Proxy215.create(Unknown Source)

Which causes this error:

Insufficient method permissions, principal=pensum.dk, ejbName=BookAuthorService, method=create, interface=LOCALHOME, requiredRoles=[Staff, User, Customer, Admin], principalRoles=null

The errors occur quite frequently, but are not caught in my (single threaded) test cases. Any help is greatly appreciated.

[milx]

I?ve looked a bit further into this, without solving the problem, but this information might cause others to respond:

When I deploy the exact same application on 4.0.3RC2, I don?t get these errors. I can se that the org.jboss.security.SecurityAssociation has been changed, adding the attribute ServerMode to the configuration of the JaasSecurityManager in the jboss-service.xml. I?ve tried setting this parameter to false, and also disabling the cache, without success.

The error seems to occur at random places involving security checks, and can easily be reproduced by using a stress test on my web application that uses the secured EJBs locally. I will try to create a test application that reproduces these errors. In the mean time any help is still very much appreciated.

Regards
Trygve

[starksm64]

The ArrayList being used is a thread local so there should not be any way for concurrent from multiple threads accessing the same list. The array list size does not make any sense though as it indicates there is a huge call stack with many run-as transitions. Post the example stress test to a jira bug report and I'll take a look at it.

http://jira.jboss.com/jira/browse/JBAS

[jpeden]

I just start the transition from 4.0.2 to 4.0.3 and ran into this issue as well. Even more interesting is one of the exceptions I ran into yesterday with the following exception:

java.lang.IndexOutOfBoundsException: Index: 14, Size: 15
 at java.util.ArrayList.RangeCheck(ArrayList.java:547)
 at java.util.ArrayList.remove(ArrayList.java:390)
 at org.jboss.security.SecurityAssociation$SubjectThreadLocalStack.pop(SecurityAssociation.java:762)
 at org.jboss.security.SecurityAssociation.popSubjectContext(SecurityAssociation.java:486)
 at org.jboss.ejb.plugins.SecurityActions$PrincipalInfoAction$2.pop(SecurityActions.java:88)
 at org.jboss.ejb.plugins.SecurityActions.popSubjectContext(SecurityActions.java:277)
 at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:159)
 at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
 at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
 at org.jboss.ejb.EntityContainer.internalInvoke(EntityContainer.java:514)
 at org.jboss.ejb.Container.invoke(Container.java:873)
 at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:415)

This happened both under JavaSE 1.5.0_03 and 1.5.0_05.

I haven't had a chance to dig any further yet today, but this definitely looks like unsynchronized access to the ArrayList to me.

Regards,

--Jeff

Jeffrey Peden
Software Architect
Newbury Networks, Inc.

[starksm64]

Ok, I think I see the problem. The default behavior of the SecurityAssociation was incorrectly changed to use InheritableThreadLocals, and the initialization of child threads is leaking the parent ArrayList value and this is resulting in unsafe usage of the list as the parent and child thread thread local are referencing the same ArrayList. This can be worked around by setting the org.jboss.security.SecurityAssociation.ThreadLocal system property to true to avoid the use of the InheritableThreadLocal in favor of simple ThreadLocals.

[starksm64]

http://jira.jboss.com/jira/browse/JBAS-2357

[milx]

Thanks for the help. I've edited the properties-service.xml to include this:

<attribute name="Properties">
 org.jboss.security.SecurityAssociation.ThreadLocal=true
</attribute>

But that does not help. Could someone please give an example of a workaround?

Regards
Trygve

[starksm64]

This property cannot be set from the properties-service.xml as that is loaded too late. Either move that service configuration into the conf/jboss-service.xml ahead of the JaasSecurityManagerService, or pass the property setting in on the vm command line using -Dorg.jboss.security.SecurityAssociation.ThreadLocal=true

[milx]

Ah, that did the trick. Again, thanks for your quick replies!
Regards
Trygve