I found a workaround for my problem. In hsqldb-service.xml I 'disabled' the HsqlDbRealm and specified username and password in there (so I cannot use the HsqlDbRealm in the login-conf.xml).
Still I wonder why my first attempt worked on Windows but throws a java.lang.SecurityException on AIX. To be more specific: method getSubject() invoked by org.jboss.resource.connectionmanager.BaseConnectionManager2 only throws an Exception when running on AIX.
Is this a bug in the new jca-framework?
Problem seems to be fixed in RC3 ;-)