Moving this from General Cache forum:
Problem: PojoCache - huge memory usage even after detaching objects. Results in outofmemory (both Heap and non-Heap). Since it keeps increasing, bumping up the memory and permgen is not a practical solution.
Summary: Running a stress test consisting of 3 objects. The lifespan of these objects is small (for the test) before they are removed from cache.
3 objects which need to stored - Order, Item, Attributes
Order has => map of Items, some long fields and an enum field.
Item => map of attributes, some long fields and an enum field.
1. I create an Order, add Item to it. Then attach Order to pojocache. (Not using Attributes at this point). I use the attach(String, Object) method.
2. The order goes thru various state changes and so does Item. After order is "Completed" state it is detached from PojoCache. All done within a transaction.
3. Since I am stress testing, I create about 4000 orders in 200 seconds, each Order takes about 10-20 seconds before it is completed and detached from PojoCache.
4. Also note that all the changes to the objects are done using a task in a ScheduledThreadPoolExecutor. One task per Order, the task is resubmitted for each state change. I do not hold the task anywhere, after Order is "Completed" it is dereferenced.
1. Heap memory contiues to grow at a steady rate peaking at abt 500 M (when the test is completed)
2. Non Heap also continues to grow - peaking at abt 112M
3. Almost a consistent slope for increase in memory
4. If I run the same test without jboss cache and using an internal hashmap the memory consumption maxes at about 15 -20Mb.
5. If I increase my load beyond a certain number I start running out of PermGen space.
1. I expect memory should not continue to increase, or is the test small that the cache does not get a chance to cleanup, I did try manual GC at the end of the test, but that did not make any difference in the memory footprint
2. So am I not doing something here? I am doing a detach of obects, anything else I need to do?
3. Also are there any know issues around Out of PermGen memory. I could not find anything on the forums.
1. Java 1.6
2. Jboss Pojo - 3.0
3. JBoss core cache - 3.2.1
4. Stand alone SE test.
I ran a small test in Yourkit and notice the top object that retains memory are ( 94MB out od 110Mb)
Objects with biggest retained size
Name Retained Size
org.jboss.aop.standalone.StandaloneClassPool 55,264,688 59 %
java.util.Hashtable 54,637,584 58 %
java.util.Hashtable$Entry 54,637,544 58 %
sun.misc.Launcher$AppClassLoader [Held by JVM] 13,694,848 15 %
java.util.Vector 13,079,760 14 %
java.lang.Object 13,079,736 14 %
org.jboss.aop.proxy.ClassProxyFactory 10,640,456 11 %
java.util.WeakHashMap 10,634,752 11 %
java.util.WeakHashMap$Entry 10,634,672 11 %
org.jboss.aop.instrument.Instrumentor 9,196,728 10 %
Things that I plan on trying in the meantime
* use strings instead of enum
* compiled aspects instead of runtime weaving
* Use 3.0.3 version of pojocache instead of 3.2.1
So here is what I have found so far. The problem is the usage of Map/HashMap that the Order and Item uses to store its children.
Order has many Items
Item has many Attributes.
Interestingly I am not even using Attributes, just the presence of that causes the test to take about 30-40% more memory.
Here is what I have observed for a test that I am running
test, Heap, PermGen, CPU, Classes loaded
With Maps for both, 468M , 112M, 8%, 11000
No Attribute Map, 285M, 64M, 5%, 7000
No Item and Attribute Map, 74M, 16M, 1%, 3300
I am going to continue to try other things, but looks like I have my culprit. The question is if this is a bug or expected??
Has anybody seen/solved this? Maybe I am just doing something wrong.
I have also moved from Core Cache version 3.1.0 ( as recommended version to be used for pojo cache 3.0)
Anand, try using JConsole for monitoring JBossCache provided MBeans for monitoring which nodes are there in the cache at each step of your test. Use very small amount of nodes for attaching and detaching, as it may cause difficulties for you to see through the Swing UI for large number of nodes.
I noticed that the internal FQNs generated by the PojoCache for keeping the actual POJO objects are not removed after detaching the objects. I am not sure if the FQNs will be removed at a later time as part of some cleanup. I put a post mentioning this here : http://www.jboss.org/index.html?module=bb&op=viewtopic&t=163083.
You can also see that the memory increase problem is not there with the Core cache. So I don't think switching to different versions of JBoss Core-cache will do good for your tests/evaluation.
I have seen the same thing, I found it is the internal (e.g. __JBossInternal__) section of the cache that is not properly removed. The nodes are null out but not removed from that cache tree when a detach() is called.
If you find a solution to this issue please share it :)