well there's not a standard hardward configuration for all enterprise application. It depends on the application. I would say I usually plan at least 2 Gig RAM for each JBoss instance but as I said it depens on the application.

Also it depends on the machine you're using if you have a 32-bit machine then you won't be able to address > then 2 Giga per process at least on Windows (on recent Linux Kernel I've read you can do a little more). If you have a process/service which graps most of the memory consider setting up a cluster so that you split the load on + nodes.

If you need tuning here's some useful information
http://wiki.jboss.org/wiki/JBossASTuningSliming
In short tune your Garbage collector + get rid of useless services

regards