You must allocate more space to your environment variables. Right click on your JBOSS short-cut ( or your prompt short-cut from where you start your JBoss) , and in the "Memory" tab,
combo "Starting environment" ( or something like this ) and select 4096. This will fix the out of memory problem.
You must declare the minimal variables for the runtime environment, they are :
JAVA_HOME -> must point to your JDK installation directory.
JBOSS_HOME -> must point to your JBoss installation directory.
PATH -> must point to %JAVA_HOME%\BIN directory and either to %JBOSS_HOME%\BIN directory.
With this settings it will work fine. ( at least here it does )
Have fun !