Well, I don't know the original reason why Hsqldb was put into JBoss originally. I can imagine some reasons: small, fast, easily embedded (i.e. not separate package with complicated config), embedded in the sense of in-vm callable.
I wanted to follow this up with what I found. We switched to the file system persistence cache, by removing hsqldb-jdbc2-service.xml and adding file-pm-service.xml to the deploy/jms directory. I plan to test mysql and others to see which is faster. For my application, the file system has fixed the strange Linux lockup bug we were having. I will pay more attention when they say something is not recommended.
I would be interested to see your mysql test results. Please post them if you run them!
What is your Linux and JVM version? If you can't get a stack trace with kill -3, then the problem is the VM not JBoss.
The File PM is okay for a few hundred messages, but doesn't work for many more. You end up running out of file descriptors. Each message becomes a file.
I would recommend either HSQL (despite what Adrian says, it's been perfectly reliable for me), or MySQL locally installed. JBoss's JMS is a lot less reliable than HSQL.
My personal experience is that file-persistance on Linux is much faster than any database storage and it can handle lots of messages.
I do not understand why sometimes one would get too many files open errors as why would the container actually need to have these files open?
We have however discovered that 3.2.3's seems to slowly use more and more memory that can be traced to JMS usage. Memory do not seem to be reclaimed.
As far as database goes. The database will use noticable percentage of your cpu and does inserts and deletes on the database which is quite slow.
On last check RollingFile persistence didn't make much performance difference, but it could probably help with the file descriptor problem.
What version of Linux are you using? There is a known problem with the NPTL library for Red Hat that causes hangups,etc. I am using Red Hat 9 and used the export LD_ASSUME_KERNEL=2.4.1 command at the top of the JBOSS run.sh.
This will force the OS to use the old Linux threads. It worked for me but then again I wasn't using any databases...just JMS.