When you say it "hangs up", is this basically a case where the lookup hangs for 60 seconds and then you get a NameNotFoundException?
If that's the case, it sounds like an intermittent connectivity issue; i.e. enough messages are able to get through that the servers are able to form a cluster, but the RPCs or RPC responses used for the cluster-wide HA-JNDI lookups are not getting through. Effect is the node doing the lookup waits 60 secs for responses that never arrive, then throws NameNotFoundException. If the servers weren't forming a cluster at all, you'd get NNFE, but it would happen quickly.
See http://wiki.jboss.org/wiki/TestingJBoss and the multicast sender/receiver tests described at http://www.jgroups.org/javagroupsnew/docs/manual/html/ch02.html#ItDoesntWork for help in tracking down multicast issues.
You can also consider using a TCP-based stack. If you want dynamic discovery of members and find multicast is working well enough for that, you can consider using MPING as your discovery protocol in a TCP-based stack. See http://wiki.jboss.org/wiki/JGroupsMPING .