The Hypersonic database is not designed to be an industrial-strength database, at least not currently. It is included with JBoss to get you up and running quickly but in production instances you would probably switch to a full DBMS, particularly when you are using the DefaultDS a lot.
However, you might be able to get short-term relief by changing the connection URL from the port-based access to an in-memory or in-process configuration. You can then also disable the HypersonicDatabase MBean.
See the hsqldb-ds.xml for more information. Note - this is for the JBoss 3.2.x distribution. There have been a few reports that the default (port-based) configuration results in many open connections in Linux when hsqldb is heavily used. This may explain the behaviour you are experiencing.
Hope it is of use.
I've seen both problems:
A huge (20+ megs) default.script file from the Hypersonic DB and also lots (hundreds to thousands) of connections back to it self