CacheListener notifications during data gravitation

[brian.stansberry]

Found an issue with BR and the AS web clustering code that's really a general problem with BR.

Issue is that both the http session repl code and clustered sso use cache listeners that interpret nodeRemoved calls as indicating a session/sso has been invalidated on another node. They respond by invalidating on their own node.

Problem is the _dataGravitationCleanup call that's part of data gravitation also generates nodeRemoved calls. But in that case, it's not appropriate for the cache listener to do anything. This causes a less than ideal behavior with session repl and breaks clustered sso.

Right now there's no way for a cache listener to discriminate between the two cases. One solution I thought of was to have TreeCache set a boolean value in a ThreadLocal in _clusteredGet and _dataGravitationCleanup. A cache listener could then check for the ThreadLocal and ignore events that arise from data gravitation.

[manik]

One solution I thought of was to have TreeCache set a boolean value in a ThreadLocal in _clusteredGet and _dataGravitationCleanup. A cache listener could then check for the ThreadLocal and ignore events that arise from data gravitation.

Kinda hacky though.

Should node removals originating from data gravitation cleanups emit notifications in the first place? I suppose so, for correctness and predictability of behaviour.

What about an additional parameter to the remove notification, to indicate whether the remove is cluster-wide (i.e., a REAL remove) or local (i.e., a data gravitation cleanup)? This would be a 2.0.0 fix though.

[ben.wang]

I don't think the distinction of LOCAL vs. REMOTE will suffice though? I thought this is problematic for remote node as well.

Maybe we should have a data gravitated event to distinguish this? But we still need to prevent the nodeRemoved event. So that may back to square one.

[brian.stansberry]

I think what Manik's suggesting isn't really LOCAL vs REMOTE but rather dataGravitation=true/false. It would get signalled on whatever node was affected by the gravitation.

That could work, and is certainly less hacky than the ThreadLocal :) But, should this be added to all notifications? In my current usage of JBC, only confusion about the nodeRemoved caused a problem. But that's just because of the way the code works; other use cases may have a similar problem with other notifications.

BTW, turns out this is not an issue for ClusteredSSO. Thinking about this made me realize BR is totally inappropriate for ClusteredSSO (I'd thought it was non-optimal but usable; it's not.)