There is an initial introduction posted by Bela on the JBoss blog.
This page deals with the issue of adding group authentication to your application. Doing so guarantees that only authorized members join a given group.
Basically, group authentication is achieved by adding AUTH to the JGroups communications stack.
AUTH is just below GMS and intercepts JOIN_REQ requests
On the client side, AUTH adds some credentials (e.g. a X.509 cert or a password) to the JOIN message
On the server side (coordinator), AUTH extracts the credentials and verifies it. If true, it passes the request up to GMS, otherwise it sends back a JOIN_RSP, which will causes an exception at the client on Channel.connect().