Having done some more digging the reason for this issue is pretty obvious... we are using DIST_ASYNC!
It was quite a while ago that we made this decision on performance grounds, and I think at the time we acknowledged that it would result in occasional dirty reads...
What happens is that when a put is done on node A it may not make it to owner node B before node B is hit with a get for the same key. I'm guessing that even when a put is done on the owner node B it sends it to itself, which is done in a separate thread and therefore another thread on node B can also get a dirty read. Does that sound correct?
With asynchronous replication, you can really see such issue - put() method does not wait for the replication (sending the write to another node), and messages sent over network are not ordered. Therefore, the read could reach the remote node before the write applies.
On the other hand, I believe that if a write is local, recording that to the data container is executed by the thread calling the put(), and therefore another read on the same node should see the value. If you can use arbitrary keys, you can implement your KeyAffinityService to do inserts (writes under the key for the first time) locally.
Hi Radim, thanks for the reply. I will do some more testing when I get some time to see if I can replicate dirty read on same node. I suspect the put and get were on the same node but not the owner node.
We don't control the keys as we use Mongo ObjectId, but I am interested in determining the owner node for a given key so we can implement affinity at the HTTP level. My idea is to set a cookie with the owner node for a given key, so that the next request comes to the owner node. If the topology changes, this is no big deal, we'd just do the remote read and then reset the cookie so user goes to new owner node next time. I think this could really help our performance by keeping most gets/puts local... does it make sense? What I don't know is if it's possible to determine the owner node(s) for a given key.
1 of 1 people found this helpful