Gravitation and POJO Cache

[jason.greene]

I was working on a solution for JBCACHE-1111 when I realized that essentially I need to wrap every cache call with:

cache_.getInvocationContext().getOptionOverrides().setForceDataGravitation(true);

This is duplicating the behavior of audoDataGravitation=true, so I am thinking perhaps it makes more sense for POJO Cache to just require that when buddy replication is enabled that autoDataGravitation is also enabled.

Thoughts?

[brian.stansberry]

Not every call; just gets.

The thing that makes autoDataGravitation=true really unworkable is that it causes a gravitation attempt any time a non-existent node is detected, whether for a read or a write. So, add a new attribute to your session == do a gravitation check, since the put creates a new node. Very non-performant. Having autoDataGravitation=false allows the app to decide when it's making a call that should result in gravitation.

Let me describe a bit about how FIELD session repl works. This seems kind of kludgy now, but anyway, here it is.

You have a node at /JSESSION/hostname/contextPath/sessionId that stores the session metadata in a couple key/value pairs. Plain cache functionality; not PojoCache.

Underneath that there is a child node for each session attribute. Those are stored/retrieved using the PojoCache API.

When a request first comes in, the session manager sets the forceDataGravitation option and does a plain cache get on /JSESSION/hostname/contextPath/sessionId. That will bring over the whole data structure, since the gravitation bring over the whole subtree under the requested Fqn. That call is the only time data gravitation is allowed. It doesn't bring the relevant _JBossInternal_, the location of which is in a separate subtree at /JSESSION/hostname/contextPath/_JBossInternal_.

Where this breaks is when PojoCache needs to traverse a PojoReference to get to the _JBossInternal_ region.

Yeah, I guess in 2.0 there are more of those calls now, since everything is now stored in _JBossInternal_.

[brian.stansberry]

For historical interest, see http://jira.jboss.com/jira/browse/JBCACHE-669. Also, there's mention on that JIRA of a org.jboss.cache.buddyreplication.aop.BuddyReplicationFailoverTest. I don't see any repackaged version of that in the current code base.

[jason.greene]

I would need to handle puts as well, since I can't predict how someone is using POJO Cache. They could retrieve an object, wait for some period time (during which the data has been gravitated to another node), then trigger an update which would fail.

Now I could provide some kind of options API, but the problem is that since the user is using standard POJO calls it is not always clear what such an operation involves.

-Jason

[jason.greene]

I would need to handle puts as well, since I can't predict how someone is using POJO Cache. They could retrieve an object, wait for some period time (during which the data has been gravitated to another node), then trigger an update which would fail.

Although, in order to solve another problem, I have to do a read before almost every put to validate that the pojo hasn't been detached by a remote node (see JBCACHE-1057). So just forcing gravitation on reads will work, although the behavior and performance will match autoDataGravitation=true.

I guess a strong reason against requiring autoDataGravitation is that like in your case, you might want to mix cache and pojocache calls in the same cache instance.

-Jason

[brian.stansberry]

They could retrieve an object, wait for some period time (during which the data has been gravitated to another node), then trigger an update which would fail.

You describe a somewhat broken usage. Let me preface this by saying buddy replication is an advanced feature, not for the faint of heart, because it requires a lot of engineering effort on the part the writer of the app to ensure that the data is truly owned by one and only one node in the cluster. It would be great if that weren't the case, but the cost of not having it be the case is terrible performance due to constant gravitation calls.

So, getting an object from the cache, holding a ref to it, allowing a concurrent request to go to another node that gravitates the cache tree, and then using the original object without re-fetching it from the cache -- that means the app is not properly ensuring ownership of the data.

[brian.stansberry]

Is it possible to check if the forceDataGravitation option is set when the initial call to the PojoCache API is made, and then preserve that value through all the internal cache calls needed to construct the pojo?

I guess that's what you meant by adding an option API. :)

[jason.greene]

You describe a somewhat broken usage. Let me preface this by saying buddy replication is an advanced feature, not for the faint of heart, because it requires a lot of engineering effort on the part the writer of the app to ensure that the data is truly owned by one and only one node in the cluster. It would be great if that weren't the case, but the cost of not having it be the case is terrible performance due to constant gravitation calls.

Ok This is a good point that gravitation needs to be manually controlled. So POJO cache should not do anything automatic here.

-Jason

[jason.greene]

Is it possible to check if the forceDataGravitation option is set when the initial call to the PojoCache API is made, and then preserve that value through all the internal cache calls needed to construct the pojo?

Yes although since references are resolved lazily (and per traversal since they could change), I would have to tag the object to always use the flag. I would limit the calls to just get calls on an internal node, but there would still be one such call per pojo operation.

I could also build an API that would allow you to control the the lifespan of the gravitation force.

I guess that's what you meant by adding an option API. :)

Yes, and alternatively there could be some kind of block form, to allow you to specify how long to do this for:

PojoCacheSession session = pojoCache.startSession();
session.setForceGravitation(true);
pojo.getBlah().getFoo();
session.stop();
pojo.setSimple();

-Jason

[brian.stansberry]

For FIELD repl I'd basically have to start and stop a PojoCacheSession on the way in and out for each request. Certainly doable.

I wonder though if the overhead of managing all this exceeds that created by just always setting the option internal before resolving the reference. (Of course don't set it if BR is disabled or autoDataGravitation=true.) Besides the work of setting the option, here's what's wasted if a call hits the DataGravitatorInterceptor for a node that doesn't need gravitation:

if (isGravitationEnabled(ctx) && MethodDeclarations.isGetMethod(m.getMethodId()))
{
 // test that the Fqn being requested exists locally in the cache.
 Fqn fqn = extractFqn(m.getMethodId(), m.getArgs());
 if (log.isTraceEnabled()) log.trace("Checking local existence of fqn " + fqn);
 if (BuddyManager.isBackupFqn(fqn))
 {
 log.info("Is call for a backup Fqn, not performing any gravitation. Direct calls on internal backup nodes are *not* supported.");
 }
 else
 {
 if (cache.peek(fqn, false) == null)

One thing I notice here is my statement last night about gravitation occuring on a put seems incorrect (see first if statement). I don't know if something changed, or if my year old memory of the issues I was seeing with BR when used for session repl is faulty. There were definitely problems related to greatly excessive gravitation calls that were solved by setting autoDataGravitation=false and using the option. I'll have a look at the old forum posts; IIRC Manik and I sorted it out on the forums.

[brian.stansberry]

My memory wasn't faulty. The test that limits gravitation only to reads was added in 1.4.1.SP3, rev 1.16.2.4, on 2007/03/16. Commit comment doesn't really describe the issue that led to the change.

[brian.stansberry]

FYI, here are a couple of the main relevant threads I found from last year:

http://www.jboss.com/index.html?module=bb&op=viewtopic&t=85462

http://www.jboss.com/index.html?module=bb&op=viewtopic&t=83772

The first one has a discussion at the end re: the proper scope of ref sharing with a POJO attribute in a web session. Basically, pojos for a webapp are stored at /JESSION/hostname/contextpath/_JBossInternal_. So refs can be shared across sessions. With BR, sharing across sessions is not a good idea, as different cluster nodes can own the sessions, and thus fight over ownership of the shared pojo. But, we don't enforce that restriction.

One possibility is to enforce that restriction by storing pojos at /JESSION/hostname/contextpath/sessionId/_JBossInternal_. If we did that, my initial plain cache get of /JESSION/hostname/contextpath/sessionId/ will cause gravitation of everything, and the problem goes away.

(I realize the above is very FIELD-repl use case oriented; not generic. I'm mostly just thinking out loud and doing a bit of a data dump here.)

[jason.greene]

I had been considering moving __JBossInternal__ to be an adjacent node to the initial referring node.

For example attach("a/b/c/User", user);

would result in:
/a
/b
/c
/User
/__JBossInternal__

The problem is, of course, that multiply referenced nodes could cross, so in your case shared references between sessions would be a problem.

-Jason

[jason.greene]

I wonder though if the overhead of managing all this exceeds that created by just always setting the option internal before resolving the reference.

I could definitely do this for resolution of internal references, you would still have to do a get on the root of your subtree with force gravitation first (as you are doing now).

-Jason

[jason.greene]

The solution that Brian and I have agreed on is that I add an overloaded find method with EnumSet. It will support one option atm, FORCE_GRAVITATE and will only affect the find call. Internal references will always force gravitation on resolution.