-
1. Re: Distributed cache with TTL and "numOwners" > 1
alexmak Jan 16, 2013 8:47 AM (in response to alexmak)I'm experiencing next behavior of infinispan (with cache configured as described in previous post)
I have cluster of 2 nodes,
each node has Cache configured with distribution, 2 copies (hash numOwners="2") and expiration after 30 minutes of inactivity (maxIdle="1800000").
I am creating an entry at node1
node1.get(key) -> value
node2.get(key) -> value
So, both nodes have entry for the key (because of numOwners="2").
Then, I am working only with node1.
After ~30+ minutes, entry on node2 gets expired and removed from cache (I've enabled trace logs, so I see when entries were expired).
And then intresting things start to happen...
node1 still has the entry, so whenever I access node1 - I see the entry,
but node2 does not have entry anymore - so call to node2.get(key) -> null
At this point Infinispan cluster behaves inconsistently.
First of all - condition numOwners="2" does not hold.
Second - result of operation depends on the node, has been accessed
Looks like Infinispan does not try to check remote nodes, if Hash function says the key is local to the current node but the node does not have that entry anymore.
2013-01-16 14:45:27,993 [qtp640494717-82] TRACE o.i.i.DistributionInterceptor [:] - Not doing a remote get for key eb490955-a105-4216-8ebc-5fc221aa1473 since entry is mapped to current node (ALEXM-1481), or is in L1. Owners are [ALEXM-1481, ALEXM-59753]
Looks like a bug.
Any thoughts?
-
2. Re: Distributed cache with TTL and "numOwners" > 1
alexgad Feb 1, 2013 10:55 AM (in response to alexmak)Did you ever get an answer or solution to this? I need to evaluate Infinispan as a solution for an upcomming project and if this is the way it works, I'd have to cross it off the list immediately as data inconsistency of this type would not be acceptable for our needs.
Thanks.
-
3. Re: Distributed cache with TTL and "numOwners" > 1
dan.berindei Feb 1, 2013 3:27 PM (in response to alexgad)Yes, this is how Infinispan works. The assumption is that you wouldn't set a lifespan for a maxIdle for an entry unless it would be easy for you to re-create that entry once it's been evicted, so not returning an entry from the cache is just as good as having an up-to-date copy from a consistence perspective. It looks like that assumption doesn't hold for your use case.
You can still "revive" the expired entry on the second owner, but you have to configure a ClusterCacheLoader (see Configuration.loaders().addClusterCacheLoader(); even better, write your own cache loader, because ClusterCacheLoader was written with invalidation caches in mind, so it asks every member of the cluster for the key instead of asking only the other owners). Even so, the entry is only revived on demand, so you don't get the high availability.
You could also use distributed task execution to force a local get on the second owner as well, and that will keep the entry alive, but that doesn't sound very convenient.
The question of cluster-wide eviction has come up with size-based eviction as well. But I guess the standard explanation was enough, because I searched for eviction issues in JIRA and I didn't find anything related to this. Perhaps you can create an issue in JIRA yourself, describing your scenario? It does sound like something that we could support in Infinispan without requiring so much fussing around (if not with the default settings).
-
4. Re: Distributed cache with TTL and "numOwners" > 1
alexgad Feb 2, 2013 4:59 PM (in response to dan.berindei)Dan, thanks for the response. I think I misunderstood how item expiration works and was confusing TTL with MAXIDLE. I assume that if TTL was set instead of MAXIDLE, the item would have been expired on both nodes. Sol I can understand it better, can I ask some additional questions:
1) If I make a request to the cluster and not to a specific node, I will get the latest version of the cached item as long as at least one version exists and it will not get me the NULL item that was expired since another value exists on another node.
2) if I have numowners set to 2 and the item expires on one node and I then UPDATE the value of the item in the cluster, one node will have the value updated and the other will have it created since the value no longer exists. In any event, I would not have to worry about getting a stale item that was older than my new updated item. Is this assumption correct?
Thanks.
-
5. Re: Distributed cache with TTL and "numOwners" > 1
dan.berindei Mar 1, 2013 2:47 AM (in response to alexgad)alexgad wrote:
[...] I assume that if TTL was set instead of MAXIDLE, the item would have been expired on both nodes. [...]
Yes, if you use TTL/lifespan then the item should expire at the same time on all the owners (createTime/updateTime + TTL).
1) If I make a request to the cluster and not to a specific node, I will get the latest version of the cached item as long as at least one version exists and it will not get me the NULL item that was expired since another value exists on another node.
I'm not sure what you mean by "make a request to the cluster".
If you call get() from an owner of that key (as determined by the consistent hash), then it's not going to look for that key on other nodes. If you call get() from a non-owner, then it's going to ask all the owners for the value, so it will return null only if it has expired on all the owners. If you use HotRod, then you're always going to hit an owner of the key, and it's not going to check with the other owners.
Like I said, you can add a ClusterCacheLoader interceptor, but then all your requests will look first on the local node, and then on all the other nodes.
2) if I have numowners set to 2 and the item expires on one node and I then UPDATE the value of the item in the cluster, one node will have the value updated and the other will have it created since the value no longer exists. In any event, I would not have to worry about getting a stale item that was older than my new updated item. Is this assumption correct?
Yes, that is correct. The value will be created or updated on all the owners.