JBoss has an inherit problem where if you restart the server with too many messages in the DB (does not matter which one) it won't come up. (You get out of memory errors.) There was a patch for this (lazy-loading queues) but had some flaws.
* Set the queue size to a maximum (no. of messages, total size ...)
Take a look at the JMX console for the queues. (You can limit by number, but not by size)
* Remove old messages automatically (timeout) without changes on the JMS clients
You can add a message interceptor. Download the source from CVS. There is a tracing interceptor you can adopt to set the expiration header.
* Use the DEFRAG function from HSQLDB
1.8, at least the one bundled with JBoss 4.0.3 automatically defrags. You can increase the size to 8GB as well.
Sorry for the late reply. Thank you very much for the great answers. Really helpful!