4 Replies Latest reply on Mar 7, 2014 8:17 AM by pruivo

    Question about PutKeyValueCommands, with delta flag set, when ownership changes / possible data loss

    thospfuller
    thospfuller Mar 4, 2014 7:21 PM

      Hi Folks,

       

      I'm working with Infinispan 5.2.3 -- here's a question I have about a potential data loss issue:

       

      We have a three node test cluster: node A, B, and C; initially only nodes A & B are running. Configuration is dist with numOwners set to two with transaction locking mode set to pessimistic.

       

      The cache that's running on A & B have some entries which are being updated, and the values implement DeltaAware, so the changes are causing PutKeyValueCommand(s) to be issued with the delta flag set (PKVCd).

       

      Node C joins the cluster while the entries are being updated, there's a rebalance, and some of the entries are moved from where they reside to the new node.

       

      So my question is simply this: what's supposed to happen to those PKVCd that are issued once the ownership has changed? Right now it looks like it's being ignored, which means the change is not applied, and hence we have data loss.

       

      For testing purposes we have a short script that iterates over a list of objects by:

      1.) getting each one from the cache

      2.) modifying it

      3.) putting it back, which causes a PKVCd.

       

      I can demonstrate data loss with this on a fairly consistent basis, though I've had to add some logic to the StateConsumerImpl to slow down the rebalancing process so as to exacerbate this issue.

       

      If this was a problem in 5.2.3 and has since been fixed, can you please refer me to the appropriate Jira? (if you know)

       

      Thanks for your help,

       

      Tom

      • 108 Views
      • Tags: