Hi, we've been using Infinispan for a couple of years -- still on version 6, thinking of upgrading to 7.x.
We have a balanced workload of read and writes with a number of different caches but one main one for long-lived website user sessions. We started with distribution async and had some stability problems in EC2 we could never really pin down, but it was related to the amount of traffic between nodes at peak traffic (our load is not huge - 200 or so read+writes per second). It may have been due to occasional EC2 network degredation. We also tried with invalidation async but gave the same occasional issues which would bring the cluster down -- we couldn't live with this but didn't have the resources to find the exact problem. We ended up reading/writing direct to MongoDB which has been giving acceptable performance. We were always using MongoDB as our persistent object store behind infinispan in any case.
We'd like to look again and see if we can take some of the load off MongoDB. One approach we considered previously is to run all the ISPN nodes in local mode (still with write-through to MongoDB), and to use the owner node to implement session affinity, such that once a session is allocated a node requests are pinned to that node while it remains available.
I think this approach would work well for our load profile and would remove the majority of chatter between nodes, and I know we can detemine the node using getDistributionManager().getConsistentHash().locatePrimaryOwner.
My main question is what happens on topology change. If a node is removed from the cluster and added back in again:
- Will it become the owner again for keys it previously owned?
- Will it automatically invalidate these keys (because data may have been changed by another node in its absence)?
Any help much appreciated.
Alfie.
