Infinispan write latency is high

[sanpingz]

Hi all,

I configured Infinispan as sync distribution mode, accessed remote cache by HotRod client. I found write latency is 2~3 times higher than read latency, can anyone help to explain it is a normal case or some wrong?

[rvansa]

If you're using multi-node cluster, this can be expected. Reads are served directly from the server you contact via HotRod, while the write has to be replicated on the backup nodes. Also, with concurrent access, locks are used and that can limit the latency.

[sanpingz]

Thanks Radim, it's reasonable that write will take longer time than read, which is the sync case.

I also tested async distribution mode to improve write latency. As expected, write latency is improved, but the read latency got worse than sync mode, I suppose async just works for write, why read latency get high? Does Infinispan read all replicas for consistency consideration in async mode?

[rvansa]

No, reads behave in the same way with both sync and async replication. If the reads are slower (how much, btw.?), it's because the cluster usage (threads occupation, context switching...) changed in some way- maybe you execute more writes in your test, so the cluster is more busy and handling reads takes longer. Check your CPU, memory and GC stats as well.

[nadirx]

Does Infinispan read all replicas for consistency consideration in async mode?

Infinispan is strongly consistent, so it doesn't have to do that. Only the primary owner is queried for the data.

[sanpingz]

Thanks Radim, finally I found it was caused by some network issues.

In addition, if my understanding is right, async distribution mode means async replication from contact node to other nodes, right?

[sanpingz]

From user guide, it said read request will be sent to all data owners in parallel and first returning result is used. As my understanding, all the owners have chance to serve query, is my understanding right? So it is possible to cause data inconsistent in async mode?

[rvansa]

With async replication, it is possible that after you call cache.put(k, v2), cache.get(k) will return v1. Eventually, all owners will contain the same value and updates will occur in the same order, but you don't know when.