mssql-service.xml uses Driver-based connections, with local transactions only. The jca wrapper for these in jboss 3 is better than the xa wrapper, it provides things like autocommit outside of jta transactions.
mssql-xa-service.xml uses XADataSource based connections. If you are using more than one transactional resource at once (such as a db connection + jms messaging) in one jta transaction, this will give you ACID transactional properties using the xa 2pc protocol. However, these seem to be harder to set up and the jca wrapper for them in jboss 3 kind of sucks.
At the moment I don't require transactions over different data sources so I guess I will stick with the mssql-service.xml