what are all jvm options you run jvm with?
Sorry....I didn't get your question exactly.... This is a dedicated server..running only this application....Java is using only by this jboss server....
What you should provide is how you set the JVM options for the JBossAS.
To see all given options you might check boot.log or server.log, here the start will show the options.
Other way is to use JConsole to see the options.
I recommend to enable GC logging which can help to see whether you run into a fullGC.
Also take a few thread dumps (use jstack - which is provided with the JDK) and check for deadlocks or busy threads.
The heap error smells like a GC issue, please post the related messages as well.