There is a quick-start guide, it's available at sourceforge along with the rest of the JBoss stuff. (PDF). The "real" JBoss docs you'll have to pay for.
Basically, find the appropriate template configuration file in
...jboss-3.2.1\docs\examples\jca, for instance mysql-ds.xml
Modify (a copy of) this file to suit your needs, especially the connection credentials, and give it another (JNDI) name than DefaultDS (which already exist), for instance MyMySQLDS, and rename it (the xml.file) to for instance my-mysql-datasource.xml.
When you want to activate the new connection, just drop the file in the deploy directory of your server. Now you can use it, for instance by specifying that your EJBS should use MyMySQL in you application specific jbosscmp-jdbc.xml.
You also have to have the jdbc-driver jar file available in .../jboss/server/default/lib. While you can hot-deploy the connection itself by moving the xml-file around, the driver jar must be available when Jboss starts (if I'm not mistaken).
While I don't really know whats best to use of mySQL and MS SQL (I guess the latter is more feature-complete ...) I've the impression that jdbc-drivers for MS SQL really, really sucks, or costs a lot of money, while Connector/J 3 for mySQL is really stable and performs well. (http://www.eweek.com/article2/0,3959,293,00.asp)