8 Replies Latest reply on Feb 17, 2006 1:22 AM by genman

    Eviction Using Too Much Memory?

    drosenbaum

      Hi,

      I am heap profiling my application that uses JBoss Cache 1.2.4SP1 and am finding that the eviction seems to be using quite a bit of memory. In particular, here are snippets of my hprof output:

      SITES BEGIN (ordered by live bytes) Fri Feb 10 09:54:18 2006
       percent live alloc'ed stack class
       rank self accum bytes objs bytes objs trace name
       1 17.58% 17.58% 13600272 17 13600272 17 49313 java.lang.Object
       2 4.39% 21.97% 3400272 17 3400272 17 49300 java.lang.Object
      
      TRACE 49313:
       EDU.oswego.cs.dl.util.concurrent.BoundedBuffer.<init>(BoundedBuffer.java:46)
       org.jboss.cache.eviction.Region.createQueue(Region.java:62)
       org.jboss.cache.eviction.Region.<init>(Region.java:70)
       org.jboss.cache.eviction.RegionManager.createRegion(RegionManager.java:61)
      
      TRACE 49300:
       EDU.oswego.cs.dl.util.concurrent.BoundedBuffer.<init>(BoundedBuffer.java:46)
       org.jboss.cache.eviction.LRUAlgorithm.<init>(LRUAlgorithm.java:38)
       org.jboss.cache.eviction.LRUPolicy.getEvictionAlgorithm(LRUPolicy.java:122)
       org.jboss.cache.eviction.LRUPolicy.configure(LRUPolicy.java:225)
      


      As you could see, in the first trace, the buffer created by the eviction Region code is allocating more than 13M of memory just for the queue. Digging further and looking at the source for org.jboss.cache.eviction.Region and EDU.oswego.cs.dl.util.concurrent.BoundedBuffer I see it is creating an Object[] array with 200000 elements!

      In the 2nd trace the LRUAlgorithm class is using more than 3 megs of memory, when it allocates an Object[] array of 50000 elements.

      Is eviction expected to use so much memory? Might I also suggest to change the implementation to use some kind of data structure that grows and shrinks as necessary so it would not consistently use so much memory.

      Or do I somehow have eviction misconfigured? Below is a snippet of my config file:

      <attribute name="EvictionPolicyClass">org.jboss.cache.eviction.LRUPolicy</attribute>
      
      <attribute name="EvictionPolicyConfig">
       <config>
       <attribute name="wakeUpIntervalSeconds">30</attribute>
       <!-- Cache wide default -->
       <region name="/_default_">
       <attribute name="maxNodes">10000</attribute>
       <attribute name="timeToLiveSeconds">600</attribute>
       </region>
      
       <region name="//net/sf/hibernate/cache/StandardQueryCache">
       <attribute name="maxNodes">10000</attribute>
       <attribute name="timeToLiveSeconds">600</attribute>
       </region>
      
       </config>
      </attribute>
      


      Thank you,
      Daniel

        • 1. Re: Eviction Using Too Much Memory?

          Daniel,

          TO answer your question. Yes, the queue size is too rigid. We do have a Jira task to change that actually. But the total memory used by eviction should also depends on your wakeupInterval. During your load test, if you eviction thread wakes too slowly, you will see the node event accumulated in the queue and therefore causing the big memory heap.

          Can you try to tune the interval again?

          THanks,

          -Ben

          • 2. Re: Eviction Using Too Much Memory?
            drosenbaum

            Ben,

            Thanks for your reply. The numbers in my profiling here is only the amount of memory used for the Object[] array with empty slots, I do not think it includes any objects actually referenced by the array itself. In other words, it is 13600272 bytes just for having an Object[] array of 200000 elements, even if all those elements are null.

            As you could see from the stack trace where it is allocated:

             EDU.oswego.cs.dl.util.concurrent.BoundedBuffer.<init>(BoundedBuffer.java:46)
             org.jboss.cache.eviction.Region.createQueue(Region.java:62)
            
            


            The code at Region.java at line 62 is
             nodeEventQueue_ = new BoundedBuffer(RegionManager.CAPACITY);
            


            (RegionManager.CAPACITY is 200000 in RegionManager.java)

            and line 46 at BoundedBuffer.java is
            array_ = new Object[capacity];
            



            That is where the size of 13600272 bytes is coming from, just from the empty array of new Object[capacity] but not considering any elements actually in it.

            (Unless if I am misinterpreting the profiler output?)

            Thanks,
            Daniel

            • 3. Re: Eviction Using Too Much Memory?
              drosenbaum

              Oh yeah, I forgot to mention that I took this heap dump when the app was initialized right after Hibernate was finished configuring. Nothing was even placed in the cache yet at this point.

              • 4. Re: Eviction Using Too Much Memory?

                I just created the following simple unit test:
                public class MemoryTest extends TestCase
                {

                public void test() throws Exception
                {
                BoundedBuffer buffer = new BoundedBuffer(200000);

                Object[] a = new Object[200000];
                a[0] = "foo";
                a[100000] = "foo2";

                buffer.put("abc");
                buffer.put("abcdefg");
                }
                }

                And ran that through Jprofiler. These are the top objects consuming memory in Jprofiler:

                Java.lang.String - 1516 instances - 36384 bytes
                Char[] - 1489 instances - 197,144 bytes
                Short[] - 740 instances - 45,392 bytes
                Int[] - 669 instances - 57,984 bytes
                Byte[] - 635 instances - 170,736 bytes
                Java.lang.Class - 565 instances - 192,928 bytes
                [] - 455 instances - 1,623,680 bytes

                Nothing else is noteworthy. Can you try running the cut&pasted unit test and see what kind of results you get?

                • 5. Re: Eviction Using Too Much Memory?
                  drosenbaum

                  Unfortunatly I don't have access to JProfiler. However I notice that a new Object[] array seems to be allocated for each Region defined, so it is not sufficient to create a test case that allocates only one array. My application has eviction policies on over 50 different regions. Might this be the problem, that I have so many regions that have settings on them? Is the LRUPolicy not designed to have so many regions?

                  • 6. Re: Eviction Using Too Much Memory?

                    Ok, that can be it then. Although there is no region limit, I don't recommend using that many regions. Eviction timer thread would have to traverse each region, that would probably be slow.

                    Why do you need this kind of fine granularity though?

                    -Ben

                    • 7. Re: Eviction Using Too Much Memory?
                      drosenbaum

                      My system has many different types of data and I need them to expire at various times. Some things I need to expire in 10 minutes, other things after one hour, others after 12 hours. The objects are also not grouped as packages that the parent can have a policy, but all of my hibernate POs are placed in the same package, so each PO type needs its own eviction policy entry.

                      • 8. Re: Eviction Using Too Much Memory?
                        genman

                        Changing the

                        BoundedBuffer
                        to this:
                         nodeEventQueue_ = new BoundedLinkedQueue(RegionManager.CAPACITY);
                        


                        should reduce memory. Probably Region should allow the Channel implementation to be configurable.