Which JBoss 3.0 are you running?
Quite a few memory leaks have been fixed since 3.0.0.
What's the definition of too many threads?
This may happens if your application creates lots of threads. The thread management in Linux is not as good as other O.S. Check total number of java thread created when it gives OutOfMemoryError.