[JBCACHE-669] Enable data gravitation for PojoCache internal

[brian.stansberry]

Please see http://jira.jboss.com/jira/browse/JBCACHE-669 for the underlying issue.

The problem arises because PojoCache bypassess the interceptor stack when doing get() calls on the cache (not sure why, prob an optimization). Therefore no data gravitation occurs within any PojoCache API call.

Cleanest solution I see is to override the optimization and do a regular get through the interceptor stack in two specific situations. This get should have an Option with setForceDataGravitation=true set in order to ensure a gravitation attempt is made. The two situations are:

1) In InternalDelegate.getRefFqnFromAlias(). This is the call that actually tries to access a node in /_JBossInternal_/_RefMap_. We need to gravitate the desired node in this situation.

2) In InternalDelegate._getAopInstance(). This is the first call made within getObject(). All calls to find pojos (whether initiated by user code or internally as an object graph is walked) go through getObject(), so doing gravitation here will ensure that any pojos are gravitated.

I implemented this by overloading InternalDelegate.get() as follows:

 protected Object get(Fqn fqn, Object key, boolean gravitate) throws CacheException
 {
 // TODO let's find a better way to decouple this.
 if (gravitate && cache_.getBuddyManager() != null)
 {
 return cache_.get(fqn, key, gravitateOption_);
 }
 else if (cache_.getCacheLoader() != null || cache_.getBuddyManager() != null)
 {
 // We have cache loader, we can't get it directly from the local get.
 return cache_.get(fqn, key, skipLockOption_);
 } else
 {
 return cache_._get(fqn, key, false);
 }
 }

With this change, FIELD granularity session repl tests that were failing now pass. The unit test for JBCACHE-669 also proceeds further (although it still fails for a more complex reason, which I'll address separately.

Ben, what do you think about this fix?

[ben.wang]

Brian, let's discuss this more 1-to-1. The two issues listed in the Jira that you mentioned seems to me are all related to get bypassing the interceptor stack, am I correct?

Yes, I am doing that exactly for optimization. During the profiling (for LOCAL mode),
http://jira.jboss.com/jira/browse/JBCACHE-574,
I have found that local get/put is still expensive because it needs to go through the interceptor stack (specially Tx and Lock). This is especially the case since during each putObject call where there are many get/set. However, the idea is that we go through the interceptor stack only once to obtain lock, then the rest of get ops should be in purely "local" mode.

Now, we are still fine with get/set under putObject wrt BR. So only problem is for getObject during failover, right?

If this were the case, then either your proposal will work. Or, in getObject() call, we have a specific call to bring over the data.

[ben.wang]

So again, two issues.

1) getObject's internal get should not bypass the data gravitation interceptor initially

2) internal mapping (/_JBOSS_INTERNAL_) that needs to be gravitated as well.

For 1), we have come up with a solution. For 2) actually, this is problematic. Currently we have the concept of region-based gravitation. It is graviated whenever one part of a region is accessed. However, the design of _JBOSS_INTERNAL_ is for global namespace and therefore won't work for region-based gravitation.

Actually, a better design should be that _JBOSS_INTERNAL_ is also region-based. That is, every region will maintain an internal map for PojoCache. This assumes the object relationship, e.g., spans only the region. A good enough assumption, I'd say. Then, whenever a region is graviatated, it will gravitate the internal map as well.

If there is no objection, I'd like to incorporate this change into 1.4. Otherwise, BR won't probably work properly for http FIELD level replication.

[ben.wang]

Brian, the other question, after looking into the code is that whether initial state transfer will have any problem if we go with the region-based JBOSS_INTERNAL?

[brian.stansberry]

I want to clarify a bit the use of terms, 'cause "region" has too many meanings :-)

A 'subtree' is a portion of the cache that's rooted in a particular node.

A 'region' is a subtree for which special behaviors have been configured. Currently these are either eviction or marshalling related.

We don't have 'region-based gravitation'. We have 'subtree-based gravitation'.

For example, with HttpSession repl, the region would be:

/JSESSIONID/localhost/mywebapp

The thing that would be gravitated following failover would be a subtree such as:

/JSESSIONID/localhost/mywebapp/54321adfad98760**

In order to resolve the problem I'm seeing with shared objects and FIELD replication, there would have to be a /JSESSIONID/localhost/mywebapp/54321adfad98760**/_JBossInternal_ node -- having /JSESSIONID/localhost/mywebapp/_JBossInternal_ would not help, since that node would not be gravitated along with the rest of the session.

I don't see how PojoCache could know that /JSESSIONID/localhost/mywebapp/54321adfad98760**/ is an appropriate place to create a _JBossInternal_ node. Only JBossCacheManager understands that that's a significant Fqn.

[brian.stansberry]

To clarify a bit more on the requirements re: FIELD http session repl:

Ben, you mentioned somewhere that we shouldn't support cross-webapp shared references. Actually, with BR, even cross-session shared references will not work. Say a POJO was an attribute in two different sessions. Then the data owner fails, and the load balancer assigns the two sessions to 2 different AS instances. From now on those two AS instances will fight over the shared POJO gravitating it from one to the other as requests come in!!!

So, if people want shared objects across sessions, they can't use BR.

Given that, *for the http session repl use case* I don't need the get() to go through the interceptor stack on a getObject() call. When JBossCacheManager calls get() on the root of the session, all attributes will be gravitated.

What I do need for http session repl is for any call that accesses a node under /_JBossInternal_/_RefMap_ to go through the interceptor stack. That is, just the call from InternalDelegate.getRefFqnFromAlias().

How many such calls would there be for a given child node under _RefMap_ during a putObject()? From a brief look at the code I don't see how there would be many. You would need at least one to go through the interceptor stack in order to lock the _RefMap_ child node. (Actually, looking at it now I'm not sure any call to a _RefMap_ child node would go through the interceptor stack, and thus you don't get a RL on that node, which seems wrong).

[ben.wang]

Yeah, you are right. Come to think of it. Our concept of region is based on per deployment instead of per app instance (like per session). So even if we do it with per-region basis, it won't work either.

Now, fortunately, I still store the data (indirectFqn) under _RefMap_ with distinct sub-fqn. If we have a global data map there, then BR will simply not work becuase each node needs to access the data.

[manik]

Interesting. Do I detect a need to be able to define Fqns as 'global', so these are always replicated everywhere? Very,very kludgy...

[brian.stansberry]

To think through the kludgy idea a bit (mostly to save others doing it)....

In general, the scope of pojo sharing needs to match up with a data structure that is meant to be "owned" by a single cache. Otherwise if a POJO is shared between two different owners, the owners will keep gravitating it back and forth. So, for example, sharing a POJO within a session is fine; sharing across sessions is not.

If a POJO is shared across data owners, then it gets replicated globally, not w/ BR. Thus the owners no longer need to fight over it.

Problems:

1) How do you know the POJO is shared across data owners. The cache doesn't know what the boundaries are of the data structure that is meant to be owned by one owner.

2) How do you ensure that all replication associated with a POJO's object graph is done globally, partic. if the changes occur as part of a tx that involves nodes that are not replicated globally?

YIKES!! Too kludgy.

[brian.stansberry]

It's not clear from the above thread, so bears saying -- Ben's fixed JBCACHE-669, so shared objects will be gravitated. The last bit about the scope of sharing is a further issue beyond the basic need to be able to follow and gravitate indirect references to shared objects.

[ben.wang]

I don't think we need it now. But one thing that I can think of is the need to have a global scope node, e.g., a read-only region where everyone can see and use. BR will break down in this case.

[manik]

Hmm, SFSB and HTTP state replication is the primary use case for BR. And the (soon to be) default way of doing is this with fine grained repl, using PojoCache.

This combo should not break down otherwise it beats the point of us doing this.