In all 3 configs you're using an async cache (clsuterin/async xml element). So it might happen that you write a key to cache A then read it from B, but the async replication hasn't happened (yet). Using sync creplication should solve the problem.
I don't think this is likely, the read and write tests are separate so I can re-run the read test a few minutes later and get the same missing entries.
right, do you wait for the cluster to form, i.e. for the two nodes to join?
You might want to enable state trasfer otherwise:
Yep, I wait for the cluster to join before running anything.
The fact that all hotrod clients exhibit this issue, and raw infinispan or the memcached interface do not have the problem makes me wonder if these are edge cases in the hotrod implementation?