You can't store the user/pw in the *-ds.xml or *-service.xml files in anything but plain literal text. However, you can write a login module to get the user/pw from anywhere you want using any encryption scheme you want. You can look at the login modules in the connector source module for ideas.
Personally, I don't see the point in any scheme that stores a password in any location encrypted with a key embedded in a program. The only scheme I know of that gives any actual security would require jboss to be started by an administrator who entered some master password that is not written to disk at any point. I don't know how to set this up at the moment.