There looks to be a slight concurrency problem in ReplicationQueue in the flush method (in HEAD).
cache.callRemoteMethods(null, // send to all live nodes in the cluster
TreeCache.replicateAllMethod, new Object[]{l}, false, true, 5000);
elements.clear();
The elements.clear call is not atomic and another thread calling add between the callRemoteMethods and the clear would subsequently clear it.
I would suggest you move the clear up in the synchronized block after the clone.
public void flush() {
List l;
synchronized(elements) {
if(log.isInfoEnabled())
log.info("flush(): flushing repl queue (num elements=" + elements.size() + ")");
l=(List)elements.clone();
elements.clear();
}
try {
cache.callRemoteMethods(null, // send to all live nodes in the cluster
TreeCache.replicateAllMethod, new Object[]{l}, false, true, 5000);
}
catch(Throwable t) {
log.error("failed replicating " + l.size() + " elements in replication queue", t);
}
}
