3 Replies Latest reply on Jul 17, 2013 2:57 PM by ggs

    Evicted/passivated entries not removed from memory in remote-stored web sessions cache

    ggs

      Hi, hopefully somebody can help me out with this issue I detected a few days ago:

       

      I have two JBoss EAP 6.1 (compiled from source) instances in a domain cluster, each JBoss instance has its own infinispan remote-store storing http sessions. The infinispan server is running in the same machine for each JBoss instance, so there are two JBoss and two infinispan instances. There is no session replication in place, the caches are local. Session storage is working correctly, the problem is that the embedded infinispan is slowly using up all available memory, because the evicted/passivated session entries are not being removed from memory. After a full GC, I can see the number of org.jboss.as.web.session.SessionBasedClusteredSession instances closely matching the sum of numberOfEntries plus passivations as reported by jmx statistics. I hope I have some configuration problem, I have tried everything and I haven't been able to change this behavior.

      My web cache container configuration is:

       

      <cache-container name="web" aliases="standard-session-cache" default-cache="remote" 
                                    module="org.jboss.as.clustering.web.infinispan">
          <transport lock-timeout="60000"/>
          <local-cache name="remote" batching="true">
              <eviction strategy="LIRS" max-entries="20000"/>
              <expiration max-idle="172800000" interval="60000"/>
              <remote-store cache="remote" shared="false" passivation="true" purge="false">
                  <remote-server outbound-socket-binding="session-binding"/>
              </remote-store>
          </local-cache>
      </cache-container>
      

       

      At the infinispan server side:

       

      <cache-container name="web" default-cache="remote">
          <local-cache name="remote" start="EAGER">
              <locking isolation="NONE" acquire-timeout="30000" concurrency-level="500" striping="false"/>
              <transaction mode="NONE"/>
          </local-cache>
      </cache-container>
      

       

      I have tried enabling/disabling passivation and the entries are not removed from memory in any case.

       

      Currently I'm using:

      JBoss EAP 6.1 compiled with Oracle HotSpot 1.6.0_45-b06 64-bit (Embedded Infinispan 5.2.6)

      Infinispan server 5.3.0

      I have tried running the JBoss server with 64-bit HotSpot 1.6.0.51 and Jrockit 1.6.0.45-R28-2.7.

      CentOS 6.4 64-bit

       

      I hope someone can help me, this has been driving me crazy and I still don't know if this is the expected behavior or I'm doing something wrong.

       

      Thanks in advance.

        • 1. Re: Evicted/passivated entries not removed from memory in remote-stored web sessions cache
          ggs

          I'll provide more information about the issue in case someone can help. Below are the first 40 entries from the heap histogram after a few hours with the server up:

           

          num     #instances         #bytes  class name
          ----------------------------------------------
             1:       8302536     3580905576  [C
             2:        388146      617252664  [I
             3:      12033043      385057376  java.util.concurrent.ConcurrentHashMap$HashEntry
             4:       8881252      284200064  java.lang.String
             5:        958270      160989360  org.jboss.as.web.session.SessionBasedClusteredSession
             6:        153729      157911104  [B
             7:       2162619      148203776  [Ljava.util.Hashtable$Entry;
             8:       2188486      132477368  [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
             9:       1511487      108170568  [Ljava.lang.Object;
            10:       3281629      105012128  java.util.concurrent.locks.ReentrantLock$NonfairSync
            11:       2155779      103477392  java.util.Hashtable
            12:       4156390       99753360  java.util.concurrent.ConcurrentLinkedQueue$Node
            13:        958112       99643648  org.jboss.as.web.session.ReplicationStatistics$TimeStatistic
            14:       2188376       87535040  java.util.concurrent.ConcurrentHashMap$Segment
            15:       2385601       76339232  java.util.Hashtable$Entry
            16:       2120443       67854176  org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager$LockState
            17:       2116510       67728320  org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager$LocalLock
            18:       2204690       52912560  java.util.concurrent.ConcurrentLinkedQueue
            19:        964104       46276992  java.util.concurrent.ConcurrentHashMap
            20:        543562       44553080  [Ljava.io.Serializable;
            21:       1275221       40807072  java.util.HashMap$Entry
            22:       2398738       38379808  java.lang.Integer
            23:        459692       34727568  [Ljava.util.HashMap$Entry;
            24:       2121072       33937152  java.util.concurrent.atomic.AtomicReference
            25:       2022910       32366560  java.util.concurrent.atomic.AtomicInteger
            26:        800471       32018840  java.util.LinkedHashMap$Entry
            27:        958475       30671200  org.jboss.as.clustering.web.DistributableSessionMetadata
            28:       1277948       30670752  java.util.ArrayList
            29:        958298       30665536  java.beans.PropertyChangeSupport
            30:        958249       30663968  org.jboss.as.web.session.notification.ClusteredSessionManagementStatus
            31:        881722       28215104  java.sql.Timestamp
            32:       1136562       27277488  java.util.Arrays$ArrayList
            33:        543348       26080704  org.infinispan.container.entries.TransientMortalCacheValue
            34:        178707       25897736  <constMethodKlass>
            35:       1143550       25228944  [Lorg.jboss.invocation.Interceptor;
            36:        178707       24317432  <methodKlass>
            37:        964105       24178984  [Ljava.util.concurrent.ConcurrentHashMap$Segment;
            38:        962177       23092248  java.util.concurrent.atomic.AtomicLong
            39:        673929       21565728  org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry
            40:         19358       21522248  <constantPoolKlass>
          

           

          The number of org.jboss.as.web.session.SessionBasedClusteredSession keeps increasing as can be seen above. Something I've found strange is that the "remote" cache as declared in the web cache container is not being used. Instead, a second cache is being created, in this case called "default-host/bligoo". This is the cache actually storing the sessions, so this could suggest that this cache is not getting the eviction configuration from the "remote" cache config, but as the jconsole screenshot I attached shows, the jmx numberOfEntries in fact is always kept below the eviction max-entries limit, so this does not seem to be the problem. Passivated entries are still not being cleared from memory.

          Any ideas?

          • 2. Re: Evicted/passivated entries not removed from memory in remote-stored web sessions cache
            rhusar

            Briefly looking at the issue, have you configured max-active-sessions in JBoss Web? The default is unlimited. I suppose what you are seeing are sessions held by JBoss Web (Tomcat).

             

            Also if you look at http://www.jboss.org/j2ee/schema/jboss-web_6_0.xsd the session passivation is off by default, have you enabled it in jboss-web.xml?

             

            Anyway, the cache configuration seems fine. You specified that cache named "remote" should be used for storing web sessions.

            • 3. Re: Evicted/passivated entries not removed from memory in remote-stored web sessions cache
              ggs

              Hi Rado, you nailed it, thank you so much! I never saw those config options in the AS7 docs, I'm now seeing that info in the documentation of older JBoss versions. In fact this solved the two problems with passivation I had: objects not being cleared and the sessions were not being passivated on server shutdown. Great!

              I have a last question related to this: when I switched from local-cache to invalidation-cache for the web sessions I noticed that instead of the SessionBasedClusteredSession instances and other clustering related objects, it was using org.apache.catalina.session.StandardSession and none of the other objects like org.jboss.as.web.session.ReplicationStatistics. Do you know the reason for such behavior? Knowing this, would you recommend using local-cache or invalidation-cache in case no session replication is needed?

               

              Thanks again, your help is greatly appreciated.

               

              Gustavo