(1) The problem with merging state is that it is very dependent on the application and also, in most cases, non trivial to do right. As Brian mentioned, merging web session state is simple, but what happens if an HTTP session has a Pojo as value ? Or even worse, an entity bean which has a primary key and is persistent ? Then anyone can get a ref to that bean and change it, so we cannot assume the session is the only way to access an entity bean.

(2) Primary partition
This is generally good, as we avoid a merging because the non-primary partition(s) discard their state and re-initialize themselves from the primary partition. However, this means that all state changes that happened in the non-primary partition is discarded ! This is probably not acceptable for most applications.

(3) Primary partitions detected when they happen
If we can say that (assuming we have 5 nodes in the cluster) a primary partition needs to have 3 or more nodes and - if it has less than 3 - becomes read-only, then we avoid state changes in a non-primary partition.
We could define a primary partition to be the majority, requiring an odd number of nodes, or come up with a different *deterministic* scheme.
Problem is: if we have {A,B,C,D,E} and then 2 partitions P1={A,B,C} and P2={D,E}, and they're all connected to a DB, then changes within P1 are replicated within P1 and persisted to the DB, but *not* replicated to P2. This means that applications connected to P2 will read stale data !
In this case, the only solution I see is to shut down P2 and redirect clients if that's possible. Shutting down could mean
- Leave the cluster (P2), so D and E disconnect from the cluster
- Suspend (or close) the AJP connector (if we come in via Apache mod_jk), or *somehow* let the load balancer know that it should redirect requests to P1 and not P2 anymore. This of course requires the load balancer to have visibility to P1and P2.

My 2 cents

(3) Another solution rather than shutting down is to cease caching values and pass all requests directly on to the underlying DB. Of course, this makes only sense if we *have* a DB, otherwise shutting down is probably our own option

(1) Re: Merget state for Pojo. Callback seems most reasonable option here. Maybe we could instruct users to implement a method that looks like:

void merge(Object o)

Where Object o is merged with by the Pojo where the method is called, User's would then merge different instance attributes and call its setters which would be intercepted. We could also say that this method would only be called in the Pojo version in the primary partition (any node in the primary partition would do cos this partition would have the same state)

If the network partition resulted in N subnetworks, merge() could get a Collection of N Pojos, one Pojo for each subnetwork Pojo.

You're right, so it looks like the only solution here is to shtu down that node and fail over clients

It'd be cool if JGroups could hook into network monitoring tools and be able to be notified of network partitions or cluster split situations.

How do these tool determine whether we have a partition, or whether some members just crashed ? Nothing different from what we want to do: they need a deterministic decision, e.g. through an independent arbiter: shared file system, DB, or separate network, or through majority decision as discussed before.
In other words, those tools won't solve the problem for us, either.

Yeah, you're right on that. There's no guarantee that such independent arbiter would be available too.

"galder.zamarreno@jboss.com" wrote:
(1) Re: Merget state for Pojo. Callback seems most reasonable option here. Maybe we could instruct users to implement a method that looks like:

void merge(Object o)

I think the primary partition approach is best. Caches not in the primary partition purging their in memory state is probably the wrong path though, since as a generic solution, not all installations will be backed by shared databases.

Caches shutting down would be my preferred option. Perhaps block for a short period, hoping the network would heal, and then throw an exception after a timeout. Perhaps a specific exception - SplitBrainException or something - so that cache users such as HTTP Replication can react by forcing an HTTP response like 410 (don't know if this is possible - Brian?) such that the load balancer will treat the node as unavailable. Once the partition heals the cache is made available to requests again after performing a state transfer to come up to speed with the primary partition.

Even the impact of incorrectly identifying a primary partition is low, since at worst case, the larger partition is unresponsive while the smaller one is. I guess the real problem is more than one partition thinking it is primary. :-)

So what are you guys going to do ? The reason I brought this up is that we need to implement merging policies in *certain* apps. For example, Brian said that he thinks a simple union-merge would be sufficient for HTTP session replication (don't know the effect on Buddy Replication though). For Hibernate 2nd level cache we might go into non-operational state and so on.
I'd like to identify the cases where we *can* do something and implement them.
For the other cases, we should provide a number of policies that a user can choose from. In some cases, a user mgith also want to handle viewChange(MergeView) himself, so we need to document what can be done.

Ok, how about:

* Provide a MergeHandler configuration
* Defaults to IgnoreOnMerge, which does nothing. FlushOnMerge also provided, which flushes in-memory state on merge

But more important than this, how can an instance tell if there has been a split, and it is not in the primary partition *before* a merge takes place? E.g., in your example of {A,B,C} and {D,E}, how can instance E decide to shutdown and not process requests?