Are you creating a new session every time you're sending a message?
I'm just speaking out loud here, but isn't that the objective of JMS - to do asynchronous message processing. I don't know if the speed matters in asynchronous processing. Ofcourse you don't want the response to take ages. OutOfMemory errors sometimes show up when the java vm runs low on memory. What is your CPU utilization when you are getting the OutOfMemory errors? Could you take a thread dump to see if there are any CPU burning threads or any threads waiting on a resource?
Just my 2 cents
We have a problem with the following scenario: when the client has sent a message to the server, he gets a message in response. We have the problem that when the client sends a lot of messages in a very short period of time and in consequence the server vice versa, the server throws the following error:
16:51:43,467 ERROR [STDERR] java.lang.OutOfMemoryError: unable to create new native thread
16:51:43,469 ERROR [STDERR] at java.lang.Thread.start(Native Method)
16:51:43,469 ERROR [STDERR] at org.jboss.mq.il.uil.UILServerILService.run(UILServerILService.java:198)
16:51:43,469 ERROR [STDERR] at java.lang.Thread.run(Thread.java:534)
Which version of JBoss are you using? There is a thread leak in 3.2.3, if you are using JMS messages with "message expiration" (TTL) and are resubscribing a lot of times. For each subscription a thread will be created and never stopped.
Did you look at the queue in jmx-console, so see how many messages were pending?