In v5 org.jbpm.task.identity.UserGroupCallbackManager is responsible for that (http://grepcode.com/file/repository.jboss.org/nexus/content/repositories/releases/org.jbpm/jbpm-human-task-core/5.4.0.Final/org/jbpm/task/identity/UserGroupCallbackManager.java).
In docs (http://docs.jboss.org/jbpm/v5.4/userguide/ch.human-tasks.html) it says if you use installer you can modify $jbpm-installer-dir$/task-service/resources/org/jbpm/jbpm.usergroup.callback.properties directly to register your own callback implementation. So edit that file to add/modify property:
jbpm.usergroup.callback=org.jbpm.task.identity.DBUserGroupCallbackImpl
In your log file you should find:
UserGroupCallback registered from properties file: org.jbpm.task.identity.DBUserGroupCallbackImpl
instead of UserGroupCallback registered: org.jbpm.task.identity.DefaultUserGroupCallbackImpl (what it says now).
That is the first part - convincing jBPM to read from DB instead of default. Now we must configure to read from your DB.
If you check the source of org.jbpm.task.identity.DBUserGroupCallbackImpl (http://grepcode.com/file/repository.jboss.org/nexus/content/repositories/releases/org.jbpm/jbpm-human-task-core/5.4.0.Final/org/jbpm/task/identity/DBUserGroupCallbackImpl.java), you will see that connection and query props are read from the same properties file as callback impl so edit that file again ($jbpm-installer-dir$/task-service/resources/org/jbpm/jbpm.usergroup.callback.properties) and add
needed props:
db.ds.jndi.name=java:/mysqlDSJNDI
db.user.query=select password from test.principles where principal_id=?
db.user.roles.query=select user_role, 'Roles' from test.roles where principal_id=?
db.roles.query=select user_role, 'Roles' from test.roles where principal_id=?
Users should now be read from your DB.