Recently my system has been in the QA. (includes Stress test, Endurance test)
During the endurance test, I found that the heap space was getting exausted, and not removed after full gc.
When the heap space got close to 90%, the gabage collection was happend every 2 min, and my system was stopped reponding.
The heap space was not cleaned up, nevertheless I ended the test (no request during a few hours)
I downloaded some heap dumps, and found that the SharedLocalYieldingLockManage.LocalLock() resided much of heap space. (above 60%)
I think the session expiration doesn't operate porperly.
If the SessionBasesClusteredSession class raises the expiration signals with "LocalOnly=True", the lock would not be removed.
Addionally, I reviewed our source code and did a leakage discory test, anything was matter.
Here is my environment. If you want to see my standalone.xml, please post a reply thread.
- 8 JVM (4 machine, each 2)
- JDK : 1.6u31
- OS : Amazon Linux 64bit
- JGroups : FILE_PING (The shared storage is GlusterFS volume)
- Infinispan operation mode : L1 + Distribution / ASYNC / expiration 1min
- Session handling : web.xml (expiration every 1min)
(Originally our setting was 30 min. Beacuse the load generator was sending too many requests with new session, I lowered the threshold to 1min)
- Source code : 7.1.3 Final tag (downloaded from Github)