Same problem here. If you'll find out how to do it please drop here a note.
In JBossAS 4.2 and 5.0, the default invoker for EJB3's is the Remoting socket transport. However, Remoting doesn't support RMI/IIOP, so I guess you would have to use the RMI/IIOP invoker. Unfortunately, I don't know anything about it. Maybe you could get some help from the EJB3 users forum.