Hi all, I am the sysadmin working with Rishabh on this.
I'd like to add that during the Tomcat startup we see two SYN packets from node A to node B being dropped by the local firewall (i.e. iptables) on an apparently random port (e.g. DPT=58317), another six from node A to node B being dropped using a different apparently random port (e.g. DPT=48800), and finally two SYN packets from node B to node A (i.e. in the remote firewall, e.g. DPT=53675).
We got everything to work by following the instructions in the following document: http://community.jboss.org/wiki/RunTwoInstanceOfJBossInFedora which suggests we just allow all TCP and UDP ports between the two nodes. That is, however, not ideal from our security policy perspective!
In case anyone follows this thread, we still have not found a solution and still find this puzzling. We continue to allow all TCP/UDP but feel there must be a better solution. Unmentioned in the original request is that we have a JCS cache running as well, although none of the random-looking ports seem to be connected with that application either. I'll update this thread in the event that I find a solution.