We are running JBM 1.4.5 on JBossAS 4.2.3.GA some time now on production. As the load on our system increased recently the system became unstable as we were still using the HSQL persistence which led to OOM exceptions.
In UAT we were able to reproduce the issue with some load tests. The heap size would remain fairly constant before crashing.
As a result we switched to MySQL 5 persistence and ran the tests again (we had to alter the max_allowed_packet to 5MB). No big deal, just 1 minute of work and it would run just fine...
To our surprise the situation was actually a lot WORSE on MySQL. The server already crashed after some minutes. The memory trend was upwards from the beginning all the way up until OOM, no drop in memory usage. Doubling the capacity of the heap and re-starting the tests instantly killed the server. It seems that JBM tries to read all messages from persistence in memory trying to re-deliver them. As the messages tend to be big, it swamped the JVM instantly.
Is there something we are missing although we followed the documented procedures?