Unfortunately, unless you provide some more information, I'm not sure how anyone can help. Exactly what are you trying to do? Log in from a client application, from a servlet, or both? Is it a normal name and password login, or some customized stuff? (Had to go through the latter myself lately, and I can attest that it was about as well documented as the art of trapping yetis in an UFO.) Do you need to secure some EJB's, or just a login for a purely JSP or Servlet site? (Unless you're actually using EJB's, I say forget about JAAS completely.) Etc.
You'd be surprised how much things can differ if you change one of the details.
Either way, the first thing you'll want to understand is that there are two sides to it. A client side and a server side. Touching just one of them or trying to treat them identically is just not going to work.
The actual login will have to happen on the server. You can't trust the client to say "oh, I've authenticated myself, and you'll just have to believe me." If you make a custom login module, as per Sun's JAAS tutorials, you'll want to put it here. (Configure your login module in the auth.conf in the jboss's conf directory.)
For custom modules do not forget to set some roles yourself. (Make up a bogus role, if you don't actually need those roles.) If you use one of the standard modules, make sure each user is assigned at least one role. At the very least, the user you're testing with.
If you use EJB's, do set permissions for each EJB method. You can use "*" to catch all the methods in an EJB, but you must explicitly declare a separate permission for the create method in the Home interface. Yes, even if you can't possibly have any use for declarative security even if you wanted to, you still can't skip that part.
(Basically: If your login module seems to succeed but nevertheless you can't actually call an EJB, then yes, one of the above two paragraphs is your problem.)
Now for the client side. On the client side you don't actually authenticate anything, at least not if you're going to call an EJB. Do not attempt to use the same login module on the client side as on the server side. You'll just want that your username and password (or any other data needed for authenticating) are transmitted to the server, where the actual login will be performed. Basically your data will just be stored somewhere, completely unchecked, and attached to the next EJB call that happens.
If you need a name and password login (or if you can serialize your custom login key to a string and call it a username), you're in luck. There's a ClientLoginModule which does that attaching and sending to the server for you. It's under the "client-login" config.
If you're trying to login from a stand-alone application (e.g., from a JUnit test case), don't forget to tell Java where to find the auth.conf file you're using. Any config file which contains an entry using ClientLoginModule is just as good.
In the unlikely event you simply can't use the ClientLoginModule, you'll have to write your own client-side equivalent of it. Just like ClientLoginModule, its main function will not be to to actually check the identity (though you can do that too, if you so wish), but to set your data in the SecurityAssociation. When the next EJB call comes, it will go to the server and get authenticated by the real login module.
And finally: Use the Source, Luke ;) Don't expect to also understand the arcane magicks there, but you can still get some idea of which bit goes where.