4 Replies Latest reply on Dec 4, 2008 9:44 AM by belaban

    jbosscache/jgroups memory leak

    hgardner

      Hello,

      I am seeing a jgroups memory leak and am wondering if anyone can provide some guidance. Below is a partial jmap histogram dump showing the jgroups leak:

      468679912 990299 byte[]
      227054264 5100368 char[]
      124964040 5206835 java.lang.String
      93450888 3893787 java.util.LinkedList$Entry
      78882192 3286758 java.util.HashMap$Entry
      69918576 2913274 java.net.Inet4Address
      69918048 2913252 org.jgroups.stack.IpAddress
      42562520 1064063 java.util.HashMap
      40117048 1065311 java.util.HashMap$Entry[]
      38877520 971938 org.jgroups.Message
      38862560 971564 org.jgroups.blocks.RequestCorrelator$Header
      31101984 971937 org.jgroups.protocols.pbcast.NakAckHeader
      23460792 977533 java.util.LinkedList
      23382312 974263 java.util.concurrent.ConcurrentHashMap$HashEntry
      23326608 971942 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
      23326608 971942 java.util.concurrent.locks.ReentrantReadWriteLock


      We are using JBoss-Cache version JBossCache-1.4.1.SP6 and jgroups version 2.6.2. Any suggestions on how to work around this?

      Any help would be greatly appreciated.

        • 1. Re: jbosscache/jgroups memory leak
          hgardner

          Here is a follow up to my earlier post...

          After testing and playing with the configuration we found that repl_async w/out using the replication queue was causing the memory leak previously reported. The issue was resolved simply by changing jboss-cache-service.xml as follows:

          < attribute name="CacheMode">REPL_ASYNC< /attribute>
          < attribute name="UseReplQueue">true< /attribute>
          < attribute name="ReplQueueInterval">100< /attribute>
          < attribute name="ReplQueueMaxElements">1000< /attribute>

          < attribute name="ClusterConfig">

          ...
          <!-- Not sure if this is required but was changed in our -->
          <!-- config, added the max_bytes attribute... -->
          <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false" max_bytes="250000" />
          ...
          < /config>
          < /attribute>

          After this change where where able to process 750 events/second per node or 1500 events/second across 2 nodes for 5+ days before stopping the test. Each event either inserts or deletes in 5 separate cache regions. Before making the configuration change we where only able to process this load for 15-20 hours before running out of memory.

          Hopefully this is helpful to someone.

          • 2. Re: jbosscache/jgroups memory leak
            belaban

            Thanks for the heads up !
            Were you using FC in your stack ? FC, together with STABLE (and max_bytes set) would prevent an OOME, too.

            • 3. Re: jbosscache/jgroups memory leak
              hgardner

              No, unfortunately I missed the Flow Control option completely. Given a pbcast.STABLE defined as:

              <pbcast.STABLE desired_avg_gossip="20000" up_thread="false"
              down_thread="false" max_bytes="250000"/>

              would you say that a FC element configured as follows would be appropriate?

              <FC max_credits="1000000"
              down_thread="false"
              min_threshold="0.10"/>

              Thanks for the input!

              • 4. Re: jbosscache/jgroups memory leak
                belaban

                Yes, that's correct. I suggest take a look at one of the configs shipped with JGroups (e.g. udp.xml) as example