It's *really* hard to debug OOME, since there could be so many different causes.
Yes, possibly the use of stateful session beans without well defined conversations *could* be a cause of this - however, SFSB passivation and timeout is supposed to protect you from this. Try adjusting the timeout.
I don't see why an application should be slow because of having many session beans. It seems that the real problem is that you need to tune the association fetching strategies used by Hibernate, and possibly use some caching.