You might increase the memory for the JBoss JVM.
If you need more help please add the current JVM settings and analyze the memory consumption.
There are several thread in the JBoss forms and other Java related
+1 to Wolf
And let me add that this doesn't seem to be related specifically to clustering, but to general problem JVM running out of heap (for any other reason). If you can reproduce it specifically when using something from clustering, provide us with more details on what you are doing and how to reproduce it.