hsqldb-plugin.jar is the JMX service wrapper for the hsqldb - it is JBoss supplied code AFAIK. You can try replacing the hsqldb.jar and see if the service runs. Just be prepared to roll back in case it doesn't.
Hope that helps.
I need to do the same thing, replace hypersonic DB with newer version. I wonder if Kundur has any luck trying this? And if he can share the info how he did it?
I was able to replace the default Hypersonic DB with a later one. I had to replace the default hsqldb.jar with the new one.