Are you consuming the messages using a Message Driven Bean?
I've experienced similar symptoms with version 3.2.6 using a MDB to consume messages from a queue. My work around was to write my own consumer - I created a Managed bean that creates x instances of my consumer. I can therefore manage memory use very tightly be configuring the number of instances of the consumer are created.
Hope that's of some help.
You might want to switch to a 'real' db such as PostgreSQL or MySQL
I don't think the DB Implementation would affect what happens on the client.
Nothing more to add, the question is unanswerable as it stands.
The orignal poster doesn't even show the jvm config or any attempt
to profile the application to find out what is using the memory.