I looked again at the documentation but couldn't find a description of what is the expected behavior when a cache is configured with INVALIDATION/SYNC, shared=false and using a SingletonStore. Any pointers to what is the as-designed case?
I also changed my cache configuration: I removed the SingletonStore and made the loaders shared. I now see the cache working properly in my test cases and saw that each node is now directly reading and writing to the CacheStore. The assumption I need to validate now is whether the locks are distributed (it must be, but need to make sure) so that consistency is guaranteed in a cluster.
For a little background, I am using Infinispan to cluster a filesystem that could be Gbytes in size. I will have a REPLICATION cache with only metadata for the entire filesystem and a INVALIDATION/shared cache for the actual file contents with a sensible eviction policy.
In invalidation mode, Infinispan passes all responsibility for replicating data to the cache store, and doesn't replicate the data at all. So in your scenario, if CM2 receives a write, it needs to write this to the cache store (in your case, the filesystem) itself. It won't pass the entry to CM1 for writing. The coordiantor shouldn't matter.
The docs on shared cache stores's aren't clear, I agree, and I've added it to the list of stuff to improve in the docs.
When a cache store is marked as shared, it means that the same cache store is present on every node, so only the originating node will store the data. So I'm surprised that in your first case that CM2 didn't write through the key to the cache store. Can you debug into infinispan to see why?