Not knowing anything about toplink, I may not be giving useful comments.
In jboss, if you deploy a datasource using the jca-jdbc wrappers, transactions in the adapter will be controlled by the jta transaction manager. However, most O/R or cache tools need to know about the tm so they can use Synchronizations to flush dirty data to the db before commit.
I see 2 things that look like datasource configuration in your sample, but nothing that looks like it shows toplink where the tm is. I would imagine that the toplink docs would indicate something about how to specify where the tm is. In JBoss, you'd look up both a datasource and the tm in jndi.
The datasource will be something like java:/myDS. The transaction manager is bound to java:/TransactionManager. Use the jndiview mbean to see the exact names.