You probably need to increase the heap size given to your jvm. Here is a snippet of my run.sh:
# increase heap size to 512 meg
java $HOTSPOT $JAXP $DEBUG $MEMORY -classpath $JBOSS_CLASSPATH org.jboss.Main $@
This is a unix or cygwin configuration, the modifications to run.bat for windows are very similar.
The memory exception is most likely from a jdk bug, that is if you are using Sun's jdk 1.3.1_01. http://developer.java.sun.com/developer/bugParade/bugs/4504200.html
I had the same out of memory exception problem, after certain loads it was very persistent. (jboss 2.4.8, tomcat 3.2.3).
Upgrading to jdk 1.3.1_04 fixed it.