'Out of memory' normaly not related to performance, it can caused by different reasons (see Javadoc can be thrown if any system resource is missed).
So please post the boot.log to see the complete JVM parameter. Also a gc.log might helpful (see http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html)
BTW what JVM version do you use?
There is no patch of JBoss to increase the performance, it is a complex issue
the max heap size set for JBoss here is 512mb.
Have you tried the obvious fix of increasing the heap size? If so, how did that go?
BTW, please do not paste log log files into your post - this topic has been rendered almost completely unreadable due to the size of the web page. Instead, attach the log file to the post. I suggest that you edit your post (the one with the log text) to clean this up.