It looks like we found the problem on this one. Our load test tool creates a new session on every iteration of a test run. because of this, we create several thousand stateful session beans during a given test scenario. These don't get cleaned up until the bean cleanup thread runs which is based on the remover-period that is set to 30 minutes by default. This explains our 30 minute performance problem.
Now, I have a question about this. It seems that when this thread runs it basically takes over the app and all other threads are basically in a waiting state. Is this a known problem or expected behavior? I would expect this to be a low priority thread because its only doing cleanup work in the background. Anyone else see this?