No, JBoss AS doesn't launch another JBoss instance.
Perhaps you're seeing threads reported as processes? AFAIK, ps -ef shouldn't give you that on Solaris though.
I don't think it's threads as I've never see Solaris threads shown as processes before, and I've seen that each one of these additional processes do take memory (3Gb, as the original JBoss process), and when I stop them the memory is released.
The extra processes are seen only a few hours after JBoss is running, and not consistently.
Yeah, the threads thing was a shot in the dark. I really have no idea. Perhaps try posting on a more general forum, maybe http://www.jboss.com/index.html?module=bb&op=viewforum&f=61 and see if someone has some insight.