5 Replies Latest reply on Mar 1, 2013 2:47 AM by dan.berindei

    Distributed cache with TTL and "numOwners" > 1

    alexmak

      Hi!

       

      I would like to get some info about Infinispan Cache running in DIST mode with Entry Expiration and "numOwners" > 1

      Especially next 2 points:

      1. 1.     How Infinispan handles expiration of entries wich have copies on several nodes. Does Infinispan remove all copies of CacheEntry when it detects that Entry is expired?
      2. 2.     How Infinispan handles access to CacheEntries wich have copies on several nodes. Does Infinispan update access timestamps of ALL copies of CacheEntry if entry is accessed on one of the nodes?

       

      I wonder if I can use such setup to store time-sensitive data (e.g. User's sessions)?

      I need predictable behavior - if entry is expired on one node - it must not be available on other nodes.

       

      Here is the config.

       

      <namedCache name="VolatileSession">

      <clustering mode="distribution">

      <hash numOwners="2"/>

      <sync/>

        <l1 enabled="true" lifespan="60000"/>

      </clustering>

      <storeAsBinary enabled="true"/>

      <!-- expire after 30 munites of inactivity - 1800000

      check for expiration every minute

      -->

      <expiration maxIdle="1800000" wakeUpInterval="60000"/>

      </namedCache>

        • 1. Re: Distributed cache with TTL and "numOwners" > 1
          alexmak

          I'm experiencing next behavior of infinispan (with cache configured as described in previous post)

           

          I have cluster of 2 nodes,

          each node has Cache configured with distribution, 2 copies (hash numOwners="2") and expiration after 30 minutes of inactivity (maxIdle="1800000").

           

          I am creating an entry at node1

             node1.get(key) -> value

              node2.get(key) -> value

           

          So, both nodes have entry for the key (because of numOwners="2").

          Then, I am working only with node1.

           

          After ~30+ minutes, entry on node2 gets expired and removed from cache (I've enabled trace logs, so I see when entries were expired).

           

          And then intresting things start to happen...

           

          node1 still has the entry, so whenever I access node1 - I see the entry,

          but node2 does not have entry anymore - so call to node2.get(key) -> null

           

          At this point Infinispan cluster behaves inconsistently.

           

          First of all - condition numOwners="2" does not hold.

          Second - result of operation depends on the node, has been accessed

           

          Looks like Infinispan does not try to check remote nodes, if Hash function says the key is local to the current node but the node does not have that entry anymore.

           

          2013-01-16 14:45:27,993 [qtp640494717-82] TRACE o.i.i.DistributionInterceptor [:] - Not doing a remote get for key eb490955-a105-4216-8ebc-5fc221aa1473 since entry is mapped to current node (ALEXM-1481), or is in L1.  Owners are [ALEXM-1481, ALEXM-59753]

           

          Looks like a bug.

          Any thoughts?

          • 2. Re: Distributed cache with TTL and "numOwners" > 1
            alexgad

            Did you ever get an answer or solution to this?  I need to evaluate Infinispan as a solution for an upcomming project and if this is the way it works, I'd have to cross it off the list immediately as data inconsistency of this type would not be acceptable for our needs.

             

            Thanks.

            • 3. Re: Distributed cache with TTL and "numOwners" > 1
              dan.berindei

              Yes, this is how Infinispan works. The assumption is that you wouldn't set a lifespan for a maxIdle for an entry unless it would be easy for you to re-create that entry once it's been evicted, so not returning an entry from the cache is just as good as having an up-to-date copy from a consistence perspective. It looks like that assumption doesn't hold for your use case.

               

              You can still "revive" the expired entry on the second owner, but you have to configure a ClusterCacheLoader (see Configuration.loaders().addClusterCacheLoader(); even better, write your own cache loader, because ClusterCacheLoader was written with invalidation caches in mind, so it asks every member of the cluster for the key instead of asking only the other owners). Even so, the entry is only revived on demand, so you don't get the high availability.

               

              You could also use distributed task execution to force a local get on the second owner as well, and that will keep the entry alive, but that doesn't sound very convenient.

               

              The question of cluster-wide eviction has come up with size-based eviction as well. But I guess the standard explanation was enough, because I searched for eviction issues in JIRA and I didn't find anything related to this. Perhaps you can create an issue in JIRA yourself, describing your scenario? It does sound like something that we could support in Infinispan without requiring so much fussing around (if not with the default settings).

              • 4. Re: Distributed cache with TTL and "numOwners" > 1
                alexgad

                Dan, thanks for the response.  I think I misunderstood how item expiration works and was confusing TTL with MAXIDLE.  I assume that if TTL was set instead of MAXIDLE, the item would have been expired on both nodes.   Sol I can understand it better, can I ask some additional questions:

                 

                1) If I make a request to the cluster and not to a specific node, I will get the latest version of the cached item as long as at least one version exists and it will not get me the NULL item that was expired since another value exists on another node.

                2) if I have numowners set to 2 and the item expires on one node and I then UPDATE the value of the item in the cluster, one node will have the value updated and the other will have it created since the value no longer exists.  In any event, I would not have to worry about getting a stale item that was older than my new updated item.  Is this assumption correct?

                 

                Thanks.

                • 5. Re: Distributed cache with TTL and "numOwners" > 1
                  dan.berindei

                  alexgad wrote:

                   

                  [...] I assume that if TTL was set instead of MAXIDLE, the item would have been expired on both nodes.  [...]

                  Yes, if you use TTL/lifespan then the item should expire at the same time on all the owners (createTime/updateTime + TTL).

                   

                   

                  1) If I make a request to the cluster and not to a specific node, I will get the latest version of the cached item as long as at least one version exists and it will not get me the NULL item that was expired since another value exists on another node.

                  I'm not sure what you mean by "make a request to the cluster".

                   

                  If you call get() from an owner of that key (as determined by the consistent hash), then it's not going to look for that key on other nodes. If you call get() from a non-owner, then it's going to ask all the owners for the value, so it will return null only if it has expired on all the owners. If you use HotRod, then you're always going to hit an owner of the key, and it's not going to check with the other owners.

                   

                  Like I said, you can add a ClusterCacheLoader interceptor, but then all your requests will look first on the local node, and then on all the other nodes.

                   

                  2) if I have numowners set to 2 and the item expires on one node and I then UPDATE the value of the item in the cluster, one node will have the value updated and the other will have it created since the value no longer exists.  In any event, I would not have to worry about getting a stale item that was older than my new updated item.  Is this assumption correct?

                  Yes, that is correct. The value will be created or updated on all the owners.