Distributed mode with numOwners=1 (default is 2) should make the object to be held in single copy, and KeyAffinityService will help you to place the object on local node. You won't be able to choose the key arbitrary, though.
numOwners=1 is fine. Also KeyAffinityService looks promising however what with Topology change? From documentation it comes - When a topology change takes place the key ownership from the KeyAffinityService might change.
Can we secure that when once is object placed to its node it never will change location? Than it would be what we are looking for.
Topology change happens when a node join or leaves. What should happen if the node 'owning' that record dies, should the record die as well? Well, with numOwners=1, upon a crash you'd lose something, so it's probably the desired behaviour... Generally Infinispan is designed with not losing data in mind The case when you can actually lose data is covered by invalidation mode cache loading data from external shared source (such as DB), but there you can't read other node's data - you just load it again from DB and if you update the underlying source, you invalidate records in the other nodes.
So, I think that if you want to hack it the way you want, you need to code your own implementation of ConsistentHashFactory, and set it in configuration (not sure if that's possible through XML, but programmatically it's in clustering().hash().consistentHashFactory()). This implementation will statically assign segments to nodes, never changing owner of (used - see below) segment.
Changing number of segments is not possible in runtime AFAIK, so you should set some expectation on max cluster size (e.g. 256 nodes), set this number of segments and assign each node one segment. The segments for which there's no node could be assigned to anyone, you wouldn't use them - each node would generate keys so that they end up locally (KeyAffinityService should inspire you).
Does this make sense to you? Sorry for not providing any OOTB solution, but there's the invalidation cache. And if you manage to get that working, contributions are always welcome
Yes when node crash and data would get lost in would be ok in our scenario.
Ok I see where you are pointing me at So custom ConsistentHashFactory should be the way. I will need to look deeper in implementation but thanks for hints.