That's the idea. You can also add username and password in the mysql-service.xml itself.
Whether you use the login-conf.xml setup or specify the user/pw in your mysql-service.xml file, you should be getting a connection like this:
DataSource ds = (DataSource) new InitialContext().lookup("java:/mysqljndiname");
Connection con = ds.getConnection();
where mysqljndiname is the jndi name you specified in mysql-service.xml
Be sure you have the correct version mysql-service.xml file for your jboss version. There have been many small incompatible changes. One that came with a binary download should be ok for that version.