I'm struggling with Infinispan (8.2.x, 7.1.x) cache consistency during a massive update of value on one key.

My cache is defined like this (I tried other options and combinations but without success):

          <replicated-cache name="refCounterCache" mode="SYNC" >

                  <locking isolation="READ_COMMITTED" write-skew="false" />

                  <versioning scheme="SIMPLE" />

                  <transaction locking="PESSIMISTIC"

                             auto-commit="true"

                             complete-timeout="60000"

                             mode="NON_DURABLE_XA"

                             protocol="DEFAULT"

                             reaper-interval="30000"

                             stop-timeout="30000"/>

          </replicated-cache>

The test case is very simple. On cache defined like this:

     Cache<String, Integer> refernceCounterCache = cacheManager.getCache("refCounterCache");

In one of nodes I set initial key value for key as 1000; Later in the same time I call on every node code that 2000 times increment value for key:

IntStream.range(0, 2000).forEach(i -> {
   try {
       refernceCounterCache.merge(key, 1, (vOld, vNew) -> vNew + vOld);
   }
   catch (Exception ex) {
       log.error("Unexpected error during value merge!");
   }
   });

When I run the above lambda code on a single node cluster (one program instance) everything works as expected (value 3000).

Problem starts when I have more than two nodes. In case of 3 nodes incremented concurrently I get unexpected results e.g. 6973 instead of 7000.

Anyone know where the problem is? Maybe wrong type of cache? Maybe wrong locking options?

I tried different options of locking, versioning , transactions, distributed cache instead of replicated etc and still without success.