Hard to say, this could be caused by:
- Garbage collection. The rule of thumb is that GC stops for 1 sec per GB of heap eventually. This also depends on the OS/JVM used.
- Both thread pools are configured to discard a message when all threads are in use. This would lead to retransmission and you have 300ms as smallest retransmit time.
- Flow control: when a sender runs out of credits, it is blocked until the receiver(s) send more credits.
I suggest you enable verbose GC and see if the times when a full GC happened correlate with your hiccups.
Per your suggestion on GC, I updated my script, which disabled explicit GC, used CMS GC, and turned on GC logging. I also upgraded to Infinispan 5.1.4.FINAL, which is using JGroups 3.0.9.FINAL. I re-run the test, and the jitters are NOT related to GC.
What you mentioned about thread pools & flow control are the general rules. In my test, I have one cache instance stand by without any activities (just to support the data replication), and another instance updating 50 data objects every 2 seconds. So I don't see those 2 rules would be applicable to my test either.
I attached my simple example again with the new script. Now run the first instance using "run 1", and the second instance using "run 2 >out.log". So could you please try running the test on your Windows platform, and see if you can find out what's the problem? Thanks in advance.
cache-write-test.zip 8.6 KB