Not quite sure why there are that many connections open. If your clustering requires external connections, you might not be able to do this. But we run hsqldb without using the JDBC port.
You can configure in the hsqldb-ds.xml that it runs-inline.
Probably more of a work-around than a solution if it does work.
I checked our configuration with lsof and it does seem to show the same problems - but we don't run clustering. It's been running four weeks.
I forgot to add that you can disable the HypseronicDatabase MBean with the in-process or in-memory modes.
I've seen this too on the machine that we use JMS on. We have other (identical setup's) of jboss 3.2.1 that have the JMX running but not in use.
I'm going to make a guess here and say that there is some jboss bug between JMS and the HypseronicDatabase that causes to to keep opening connections and not closing.