There is no direct answer, but I did watch a good talk by Bela Ban on this. The context was in the mod_cluster "domains" - having domains of cluster nodes that are kind of independent from other domains (so within a domain you can have the replication you want/need, but not across domains). At the end of it, he sees 100 nodes as generally being a limit where you would want to consider splitting it into domains (ie overhead of replication gets annoying).

Unfortunately the presentation isn't public (at the moment) but hopefully that will change soon, as it was a brilliant talk.

Hi Matthias,

there are 2 types of redundancy: one's called replication and the other one distribution. Replication replicates every data to every other node in the cluster, so the cluster size is bounded by (1) the number of nodes and (2) the average data size of each node. E.g. 10 nodes with an average data size of 100MB means that every node has to store data of 1GB on average.

Distribution (as done for example in Infinispan) means that a data element is stored N times in the entire cluster. So if every data element is stored 2 times, then the average data stored at any given node is 200MB (versus the 1GB for replication).

However, if 2 nodes storing the 2 data element die at the same time, your data is gone. So, distribution is a tradeoff between data usage and data redundancy.

Bela

The cool thing in Infinispan is that different caches can be created with different Ns, and you place your data in a cache according to your reliability requirements.

Re: replication versus distribution: I guess if you're cluster is smallish and/or your data small, then I'd prefer replication because reads are way faster (because they're always local) than in DIST, where we might have to do a sync round trip if the data is not in the L1 cache.

So, for example, if you have 10 nodes and every node has 10MB worth of data, then I'd go for replication. If you have 50 nodes, and every node generates 200MB worth of data, then I'd pick distribution.

No, the load balancer has no idea about how Infinispan distributes its data. E.g. if we have {A, B, C} and N=1 and data element D is stored on B.

Say the load balancer routes a session S to A, then A will ask B for the data the first time it doesn't find it in its L1 cache. Subsequently, A will always return D immediately, unless and until D is invalidated (e.g. because it was updated).

At some point, we should brainstorm a bit about whether it makes sense to add pluggable routing (in C) to the load balancer. If we had the same consistent hashing algorithm on the load balancer (mod-cluster) as in Inifinispan, the load balancer could reduce the number of additional round trips to fetch data not in the L1 caches...

Applied to HTTP sessions, individual attributes of a session are stored on different nodes, so 'stickiness' here would be broken down to attributes, not an entire session. As an alternative, a consistent hashing algorithm could used the jsessionid as input, so all attributes of a given session would be colocated.

This needs more discussion (probably on the mod-cluster forum), I'll ping Brian ...

That's something to discuss. One could argue, as attrs of a sessions are accessed together, that they should be stored together. OTOH, if the attrs are big, then it might make more sense to distribute them. Distribution of individual attrs adds more complexity to mod-cluster though.

We'll discuss this during our Brno meeting next week.