[manik]

Ok, the way this works is:

a) By default, use EnhancedTreeCacheMarshaller, regardless of the UseRegionBasedMarshalling - fka UseMarshalling - property.

b) If running in compat mode and UseRegionBasedMarshalling is set, use TreeCacheMarshaller.

c) If running in compat mode and UseRegionBasedMarshalling is not set, don't use any marshaller (reverts to serializing all calls).

Is this how everyone sees the compatibility mode working? And for a flag, how about 'UseReallyInefficientButCompatibleMarshallingIDontSeeWhyYouWouldRUnWithThisEnabledForLongPeriodsOfTime'? :-)

[brian.stansberry]

OK, now to drive you really insane...

1.4.0 now makes heavy use of MethodCall.getId(). Not just in marshalling, but also in switch statements throughout the code.

MethodCall.getId() was introduced in 2.2.9. This means 1.4.0 requires 2.2.9 or higher.

To me, the point of interoperability between 1.4.0 and earlier releases is so 1.4.0 can go into the 4.0.x code base as the default cache. But, the default JGroups in 4.0.x will always be 2.2.7. This means 1.4.0 can *never* be the default. So, if 1.4.0 gets into 4.0.x it will have to be as an installer option.

The purpose of interoperability is to allow a rolling upgrade of a cluster. But the JG incompatibility will prevent such a rolling upgrade -- the cluster will have to be taken down (or segregated into 2 subpartitions). If they have to take down the cluster to upgrade to 2.2.9 anyway, I see little point in interoperability at the JBC level.

The only benefit of interoperability at the JBC level is if the cluster has already been upgraded to 2.2.9 or higher. If that has been done, then a rolling upgrade of JBC is possible.


So, what do people think? Do you see any point in backward interoperability? For me it's a -0, but with a nagging feeling in my gut that we'll regret it if we don't do it.


If we do, here are some implementation details:

1) Manik, I know you were joking about the flag name, but again the existing ReplicationVersion attribute can control this behavior.

2) With the MethodCall.getId() stuff, if we deserialize a MethodCall via b) or c) we need to swap out the MethodCall object via a call to MethodCallFactory.create(). Otherwise the deserialized MethodCall will not have the proper id.

3) To get the true benefit of interoperabiltiy, the replication version short should be written at the beginning of each message. The recipient then reads that short and based on it decides how to deserialize the message. If the short doesn't exist, treat that as meaning replication version < 1.4.0.

Why is this needed? Allows you to move from compat mode to efficient mode without a cluster restart.

2 node cluster with servers A and B, running JBC 1.2.3.

1) Bring down A. Upgrade to JBC 1.4.0 with replication version set to 1.2.3. A and B interoperate.
2) Bring down B. Upgrade to JBC 1.4.0 with replication version set to 1.2.3. A and B interoperate, but at a lower rate of effectiveness than necessary.
3) Bring down A. Change replication version to 1.4.0 and restart. A now starts sending out messages with the 1.4.0 short. B knows how to handle those messages. B's messages still have no short, but A knows how to handle them.
4) Bring down B. Change replication version to 1.4.0 and restart. Now all messages have the 1.4.0 short, and the cluster is operating in the more efficient 1.4.0 mode.

The byte[] used by state transfer works this way. I included the short in the byte[] because Bela told me to; didn't really think through the benefit of it until much later :-)

[manik]

OK, now to drive you really insane...

I eat insanity for breakfast! :)

MethodCall.getId() was introduced in 2.2.9. This means 1.4.0 requires 2.2.9 or higher.

Yes, this was always a point of concern for me, but this was the conclusion we arrived at on the relevant discussion thread. There is a very simple workaround: for us to subclass MethodCall in JBoss Cache, and add an ID there. This will allow us to use JGroups 2.2.7. There is no other dependency on JGroups 2.2.9, except that I would like to ship with this for performance.

2) With the MethodCall.getId() stuff, if we deserialize a MethodCall via b) or c) we need to swap out the MethodCall object via a call to MethodCallFactory.create(). Otherwise the deserialized MethodCall will not have the proper id.

Yes, this will have to happen.

3) To get the true benefit of interoperabiltiy, the replication version short should be written at the beginning of each message. The recipient then reads that short and based on it decides how to deserialize the message. If the short doesn't exist, treat that as meaning replication version < 1.4.0.

Aside from adding a further short to every message, I'm all for versioning comms. The flexibility it will allow us is huge.

[manik]

Aside from adding a further short to every message, I'm all for versioning comms. The flexibility it will allow us is huge.

Thinking about this, I think we will have problems expecting the first 2 bytes in a stream to be a short representation of a version.

In the case of trying to unmarshall data from JBC < 1.4.0, we will trip up trying to read arbitrary stuff as a short. In most cases, this will return gibberish or an exception, in which case we can safely assume it is pre-1.4.0 and use the appropriate Marshaller, but it *may* just produce a valid version short even though the stream is not of that version.

How would we deal with that case? Bubble up the exception that the marshaller will inevitably throw when trying to deal with a stream it is not designed to deal with? That completely breaks interop though.

Bela, how do you deal with this in JGroups, or have you always had the version bits in your header?

[brian.stansberry]

MethodCall.getId() was introduced in 2.2.9. This means 1.4.0 requires 2.2.9 or higher.

Yes, this was always a point of concern for me, but this was the conclusion we arrived at on the relevant discussion thread. There is a very simple workaround: for us to subclass MethodCall in JBoss Cache, and add an ID there. This will allow us to use JGroups 2.2.7. There is no other dependency on JGroups 2.2.9, except that I would like to ship with this for performance.

Want to explore the workaround in more depth. We have our own MethodCall subclass that adds the id field. For this discussion I'll call it JBCMethodCall. JBCMethodCall is redundant if we're using 2.2.9+, but so what, when we marshal the method call in 1.4.0 mode we're doing it efficiently.

If ReplicatonVersion < 1.4.0:

1) When we unmarshal the replicated MethodCall, we pull out the encapsulated Method and args and call MethodCalllFactory.create() to get a JBCMethodCall. No extra cost here -- as mentioned in previous posts we have to do this anyway.

2) When we marshal a JBCMethodCall, we create a jgroups MethodCall from its Method and args and marshal that.

Actually, I like it. Not too much overhead.

[belaban]

Very simple: if the 2 bytes match, then it is the same version number, otherwise I just discard the message (if that is configured)

[manik]

What if the first 2 bytes are gibberish - i.e., generated by a marshaller that does not expect to place a version id - but because of pure luck, happens to be 2 bytes that may *look like* a valid version id?

[belaban]

The marshaller does not expect to place a version number ? I'd say that should be pretty much mandated by every marshaller...

[manik]

Mandated by every marshaller from now on, yes. But previous marshallers - e.g., in an up and running instance of JBC 1.3x or 1.2.x - will not place a version number.

[galder.zamarreno]

I go away for a week and when I return, you have changed all the work i
did!!! shocking ;-)

It makes sense anyway and pity it didn't show up when we were first
discussing how to improve MethodCall.equals() performance.