I suppose you use a 32Bit JVM, the memory is limited there depend on the OS and JVM.
You might check with a standalone Java-app how much memory is possible.
Also you should add your JVM parameter/config here.
Do you see any OOM Exceptions in you server.log?
What mechanism are you using to start JBos AS as a service (there are three that I have used). If you are using service.bat, have you taken care of the issues where JAVA_OPtS is set in service.bat and thus ignored in run.conf.bat? (There have been numerous posts on that topic)
Have you monitored the heap usage of garbage collection data?
Have you verified what options are actually getting set for JBoss AS by checking the commandl ine used to run it? You can do this either via Task Manager (though it often cuts off most of the data) or Process Explorer from sysinternals.
I do have the 64-bit JVM installed. As I stated it runs fine with 2GB memory when run from the DOS prompt.
The JVM parm that works is set JAVA_OPTS=%JAVA_OPTS% -Xms512m -Xmx1024m
What works in DOS prompt but does not work when run as a service is: set JAVA_OPTS=%JAVA_OPTS% -Xms1024m -Xmx2048m
Parts of the code utilize a great deal of memory. I would like to set the max memory to at least 8GB and probably higher. I have seen forums where developers say they have allocated up to 16GB to JBoss. I just need to figure out how to make it work. If it hadn't worked in the DOS prompt I would blame it on my settings in the run.bat.
I am using JBossNativeWindows. It executes jbosssvc.exe and the only parm is -r JBAS42SVC. I have checked the service using Process Explorer. jbosssvc.exe executes service.bat which in turn executes run.bat. According to Process Explorer it did pick up the memory parms from run.bat. The file run.conf has different settings for memory but they are not used.
I am trying to figure out how to monitor garbage collection. Tried a couple of options that wouldn't work.
Adding these options to JAVA_OPTS where you are setting heap size should get GC data:
And another thought - what Windows account are you using for the JBoss service? I typically don't use the system account, instead I set it to an account specifically created for that service (or to my account when doing dev/test). I also make sure that the account has full rights to the JBoss AS installation directory.