Controlling state transfer behavior with BuddyReplication

[brian.stansberry]

I want to make a couple simple cleanup fixes to how state transfer behavior is controlled if BR is used.

1) If BR is being used, the standard initial state transfer triggered in startService() should be automatically disabled. Any state transfer at startup should be done as part of the formation of buddy groups.

2) Any time there is a state transfer in the context of BR, what state gets transferred is controlled by the same set of flags that we've used for the non-BR case:

If fetchInMemoryState is true, the in-memory state is transferred.

If there is a CacheLoaderManager and its fetchPersistentState property is true, persistent state is transferred.

Right now I'm always fetching in-memory state as part of buddy group formation, but I don't see any reason why the configuration options should be different than the usual case. If the buddy can retrieve state from a persistent store, it isn't requirement to transfer the in-memory copy.

Thoughts?

[manik]

I presume you're referring to a 'shared' persistent store, in which case it makes sense. In fact, with a shared persistent store, why bother with buddy replication at all?

[brian.stansberry]

I presume you're referring to a 'shared' persistent store, in which case it makes sense. In fact, with a shared persistent store, why bother with buddy replication at all?

Are you talking about this statement?

If the buddy can retrieve state from a persistent store, it isn't requirement to transfer the in-memory copy.

If so, I really meant an unshared store (you're right - BR with a shared store is silly). I meant that if all state is in a persistent store on the DataOwner, and that state is transferred to the buddy as part of a persistent state transfer, then there is no requirement to do an in-memory state transfer as well. That's a configuration choice up to the user; depends on whether they want the in-memory backup tree to be cold or hot. This is no different from the non-BR case.

[manik]

Yes, I was referring to

If the buddy can retrieve state from a persistent store, it isn't requirement to transfer the in-memory copy.

I agree with you about the non-shared case though. The in-memory state could be optional (and probably should be set to false in the BR scenario since the state transferred is backup data after all)

[manik]

Just spotted something when discussing some BR concepts with Bela last week - how would state transfer work if we are using optimistic locking? This is actually irrelevant of whether BR is used.

[brian.stansberry]

Basically, a RL is acquired on the root node of the region being transferred, and then the node is passed to this method:

private void marshallTransientState(DataNode node,
 ObjectOutputStream out) throws Exception
 {
 Map attrs;
 NodeData nd;

 // first handle the current node
 attrs=node.getData();
 if(attrs == null || attrs.size() == 0)
 nd=new NodeData(node.getFqn());
 else
 nd=new NodeData(node.getFqn(), attrs);
 out.writeObject(nd);

 // then visit the children
 Map children = node.getChildren();
 if(children == null)
 return;
 for(Iterator it=children.entrySet().iterator(); it.hasNext();) {
 Map.Entry entry = (Map.Entry) it.next();
 marshallTransientState((DataNode) entry.getValue(), out);
 }
 }

Whatever nodes are returned by node.getChildren() will be included in the transfer. You just told me workspace nodes will not be returned by that call. So, the state transfer will ignore the workspace, which IMO is correct. The state transfer recipient will get the workspace modifications as part of normal replication when the tx commits.

[manik]

Yes, so far this sounds good, but what about the other end, where state is applied?

Do you explicitly ask for write locks on the nodes, or would you pass up put() calls which will result in a workspace for the state transfer 'transaction'?

The reason I ask is that I wonder how a concurrent put() and state transfer would be handled.

[brian.stansberry]

Not sure I fully understand your question.

On the recipient side, a RL is acquired on the root node into which the state will be integrated (I think this s/b a WL -- IIRC correctly Ben and I discussed this last Sept, but I don't remember why we decided a RL is OK). Then that node's data map and children map are cleared, the data map from the first NodeData in the passed state transfer is integrated, and new Nodes are created and inserted for all subsequent NodeData objects in the passed state transfer.

There is no put() call or transaction involved.

Re: a concurrent put and state transfer, not sure exactly case you are describing:

1) Someone calls put on the cache receiving the state transfer at the same time the state transfer is on progress. This is a mistake in the application, exposing the tree to put() calls before the state transfer is done.

2) Someone calls put() on the sender while the state transfer is in progress. In that case, the workspace node will be created on the sender, will not be included in the state transfer, and will be replicated when the tx that wrapped the put() is completed.

[manik]

Makes sense. Just wanted clarification. I wasn't sure how state was applied to nodes on the recipient end, i.e., if nodes were locked manually or passed up the interceptor chain as method calls.

What you're saying makes sense though, and should be safe for both optimistic and pessimistic locking modes.