Can avoid the error with "-server" by changing the EJB/Hibernate loading sequence, i.e. by pre-loading some reference Entity objects earlier rather than on-demand.
Does JBoss do any major initialisation load testing with EJB3? I mean 20+ SQL statements on Web-App startup (i.e. in first request). Many of the EJB NamedQueries use 'join fetch'.
P.S. In the original error CompilerThread1 was trying to allocate 33,554,440 bytes, i.e. 32MB+8bytes - and then complaining about 'out of swap space?'.