There's a design document which is part of JGroups: JGroups/doc/design/AUTH. There is also an AuthTest unit test which shows how to configure/use AUTH
I've searched for those items in your response, but cannot find them, about the closest I get is at this site http://www.jgroups.org/javagroupsnew/docs/papers.html. Can you point me to where exactly those are?
I downloaded the source for JGroups, and was able to find some documentation on AUTH and ENCRYPT there, but I was left a little bit confused. I had assumed that AS clustering used JGroups already to perform clustering, but now I'm not so sure. The documentation for JGroups talks about installation of JGroups project, which leads me to believe that it is altogether a separate entity. If that is the case, how does it integrate, and how does it get used by the cluster?
Yes, the docs are part of the JGroups CVS.
Yes, JGroups is used for JBossAS clustering. The clustering software that uses JGroups is an MBean called ClusterPartition. Otherwise, all state related functionality (e.g. replication) is done on top of JBossCache, which uses the same JGroups JAR