the correct path to the JAR is "/common/lib/jbosssx.jar" (changed in JBoss 5.x) ;-)
You might enable logging of the security layer to find out the "real" error: http://www.jboss.org/community/wiki/SecurityFAQ (question 4)
thanks for the reply. Unfortunatly the file common/lib/jbosssx.jar does not contain the class RFC2617Digest.
I guess that's the problem.
this is located in "/common/lib/jbosssx-server.jar". Sorry for seeing this too late.