Is there a reason
Class#cast(Object) instead of a normal Java cast? We did some profiling of our application and this method showed up.
A related issue seems to be that
SimpleRoleGroup#roles is a ArrayList rather than a Set so every add* method has to call
java.util.ArrayList#indexOf(Object) which has linear rather than constant complexity.