Http session passivation Design

[hmesha]

Hi all,

below is a draft of the design doc for this feature. I have couple of conerns that I'd appreciate your feedback on.

Regards,

Hany.

HTTP Session passivation/activation feature design
==================================================

* configuration (tomcat/resources/tc5/tc5-cluster-passivation-service.xml):

1. eviction policy configuration
2. cache loader configuration with passivation turned on

* passivation

From JBossCache eviction policy implementation, The eviction policy is invoked based on the configured wakeup interval to process cache nodes (sessions) eviction based on the configured eviction policy attributes. The CachePassivation Interceptor will intercept the evict method calls on the nodes (sessions). It'll emit pre-passivation notification and store them in the configured passivation store (cache loader store) (filesystem or database).

* activation

When the node (session) is requested, it'll be loaded from JBossCacheManager.loadSession() which eventually calls cache.get(). It'll be intercepted by the ActivationInterceptor loading it from the passivation store (filesystem or database) and emits a post activation notification.

* Notification

The notification emitted by the TreeCache will be broadcasted to the listeners that have registered with the TreeCache. In particluar, org.jboss.web.tomcat.tc5.session.CacheListener. Therefore its passivate or activate method will be invoked to notify the session listeners that the session is about to be passivated or has been activated. First, we check if the passivation/ activation event is emitted from a non-session FQN and this node (server) in the cluster, we skip it. Otherwise:
1) In passivate, we find the local session by calling JBossCacheManager.findLocalSession(). if the session found, call its passivate method.
2) In activate, we find the session by call calling JBossCacheManager.findSession() which will call loadSession if the data is outdated or the session doesn't exist in the manager's local store.
Concern: if the session is not found or outdated during activation, JBossCacheManager.loadSession() will be called and that trigger session.activate() which called from session.initAfterLoad(). That means, potentially, we call session.activate() twice. what should we do about it?

solution:??

* passivate/activate methods
The session passivate/activate method is implemented in ClusteredSession to notify JBoss session attributes of type HttpSessionActivationListener, then it calls the StandardSession.passivate() or StandardSession.activate() to allow the catalina session to notify the catalina attributes of the type HttpSessionActivationListener as well.


* Flow control

JBossCacheManager.processExpires() is invoked from JBossManager.backgroundProcess() every 60 seconds (default) which expires idle session periodically. The logic of expire() is based on maxInactiveInterval which is set in JBossCacheManager.createSession(). In order to have a successful implementation here, we need to coordinate between backgroundProcess() and the eviction policy in use. Potentially, we have 2 different timers running here:
1) the eviction plicy which is enforced based on the wakeup interval (default 5 sec) set on the cache
2) the session time out based on maxInactiveInterval which is enforced by backgroundProcess() based on the configured value (default 60 seconds).
JBossCacheManager.processExpires() does the following:
1. findSessions()
2. iterate through the sessions
3. make sure that we're running in a transaction
4. check for the session data validity isOutdated(), which is set by the CacheListener when the node is identified to be dirty,
5. and session validity isValid(false). If the condition evalutes to true, the loadSession() is called which trigger activation. The activation will reset the last access time therefore, the session will be valid.
6. (concern) check the maxInactiveInterval by calling isValid() which in turn calls isValid(true) and if the session exceeded the maxInactiveInterval, it will be evicted from the cache, triggering passivation, and the session will be destroyed which makes it impossible to activate the session afterwords.

solution:
When the session exceeds maxInactiveInterval, should be removed not evicted from the cache. that will ensure that it'll be removed from the passivation store which will keep the passivation store in sync with cache at all time.

JBossCacheManager.stop() is invoked during AS shutdown. The first step in the stop process is to expire active sessions. When passivation is turned on, we should passivate those sessions instead of expiring them. That can be acheived by controlling the flow in JBossCacheManager.clearSessions(). if the passivation is turned on in the configuration file, then passivate sessions. Otherwise, just let the expiration take place.

[ben.wang]

Hany,

Thanks for the effort. Here are my comments.

* passivation

From JBossCache eviction policy implementation, The eviction policy is invoked based on the configured wakeup interval to process cache nodes (sessions) eviction based on the configured eviction policy attributes. The CachePassivation Interceptor will intercept the evict method calls on the nodes (sessions). It'll emit pre-passivation notification and store them in the configured passivation store (cache loader store) (filesystem or database).

We should not need to use eviction policy. Instead, we should rely on Tomcat background process to do the eivction (as you have commented later). This way, we don't need to worry about the contention.

Therefore, when Tomcat is about to expire a session, internally instead of doing local remove, we should perform "evict" instead.

[hmesha]

Ben,

Thanks for your feeback.

1) Using eviction policy will add the feature of pluggable custom passivation policies to the design.

2) What you mean by contention?

3) backgroundProcess() which calls processExpires() should continue to remove timed out sessions so when we do passivate node, we'll only passivate valid sessions. Currently, that means the session is evicted rather than removed locally. I understand that we're using evict() to avoid replication of the remove on other nodes where the session still in use. However, it shouldn't be calling evict() either since evict() when passivation is turned on will trigger passivation. I'm suggesting to introduce a new method call in TreeCache that will remove the node locally but doesn't trigger replication and processExpires() should make use of that method call.

Therefore, we don't need to add code to processExpires() to handle passivation since passivation is done automatically.

On the other hand, I see that we need to do special handling when the session manager is about to stop as you see in the last few lines of the design doc.

Cheers,

Hany

[ben.wang]

Hany,

1. Why do we need eviction ploicy in order to use passivation. All we need is "evict" method call and that will trigger the passivation, isn't it?

2. I meant background thread and eviction thread (if configured) may run into the same dataset, and therefore the contention.

3. OK. I guess how does Tomcat (non-clustered mode) do the passivation (or configured then)? I thought when the session expires, it is passivated? Or I am mistaken?

Thanks,

-Ben

[ben.wang]

Hany,

I am just curious when your implementation is completed, will we mimic what the standalone (non-clustered) notation as in here
http://tomcat.apache.org/tomcat-5.5-doc/config/manager.htm?

That is, will we have similar semantics like maxIdleSwap, etc?

Thanks,

-Benl

[hmesha]

Ben,

First, let me apologize for not responding fast enough to your questions. I was extremely busy with JBossCache and internal work here in Novell.

1) We need eviction policy to give users the ability to either use the exsiting eviction policies or write custom eviction policies to control the passivaton. For example, LRU would be a good candiate to evict the sessions (passivate) based on the time. This also will allow us to have similar semantics to tomcat's persistence manager which answers your subsequent question

will we have similar semantics like maxIdleSwap, etc?

2) Maybe what we ought to do is to have the ClusteredSession.expire(boolean, boolean) to check if the manager's cache service passivation is turned on and the eviction policy in use condition evalutes to true, evict node. Otherwise remove the session and destroy it. which in turn will remove it from the passivation store.

Thanks,

Hany

[hmesha]

Ben,

As far as the implementation completion, I already have the code in place and ready to be checked in against HEAD once the issues at hand are resolved.

Cheers,

Hany

[brian.stansberry]

A few thoughts:

1) Re: JBossCache API needed to support session management. There are 4 events that cause removal of a session from a particular cluster node's cache:

a) Passivation. evict(Fqn) with the PassivationInterceptor should do.
b) Webapp shutdown -- again evict(Fqn) w/ PassivationInterceptor.
c) Logout via call to session.invalidate(). remove(Fqn) w/ the CacheStoreInterceptor removing the session from the persistent store as well.
d) Session expiration. remove(Fqn, Option) with the Option.setCacheModeLocal(), thus ensuring removal doesn't propagate. CacheStoreInterceptor removes the session from the persistent store as well.

Hany, does this sound like it matches your needs?

2) Shared vs. unshared store. This is an issue in the case of session expiration. Sessions are independently expired by different cluster nodes. If a shared store is used, a session expiring on one node could lead to invalid removal from a shared store. This is a bit of an edge case, where the background thread on one node expires a session, while on another node a request comes in just in the nick of time to keep the session alive. Can also happen if there is read-only activity on one node that is keeping the session alive but not causing it to replicate.

A potential solution to this problem is having the CacheStoreInterceptor check for Option.isCacheModeLocal() == true. If so, AND the cache loader is shared, don't remove. Haven't really thought this through, and implementing such behavior would be in JBossCache and thus affect all JBC use cases.

Also, if the session is not removed from a shared store when it expires, it may never be removed (i.e. it expires on all nodes, and is never removed from the store). So we'd need some kind of process to come along and do a cleanup. Ugly.

My initial inclination on this is to design for an unshared store and document this. Thoughts?

3) I'm a little confused by the bit about activation and the TreeCacheListener. I think we need to establish some ground rules as to what triggers a notification. There is an important distinction between the JBossCacheManager and the cache itself. IMHO, activity solely in the cache is insufficient to trigger a notification. For example, say a session has been passivated. Then a replication message comes in from another node. If I understand it correctly, processing the replication message would lead the ActivationInterceptor to activate the session's nodes in the TreeCache. However, this shouldn't trigger an activation event; the activation occurs when the JBossCacheManager once again loads the session from the cache. This may never happen.

Same thing with passivation. I'm working on a mechanism whereby sessions that are replicated (and thus in the cache) but never accessed on a given server never get loaded by JBossCacheManager -- they exist only in the cache. If such a session is never accessed on a cluster node, there would be no activation notification on that node. If the session were later passivated to clear space in the cache, there should be no passivation notification either.

4) Configuration. A major goal I have is to simplify configuration of session replication. Currently up to 4 files can be touched by session repl config -- jboss-service.xml, server.xml, tc5-cluster-service.xml and jboss-web.xml. Almost everything we currently configure is actually changeable per webapp. I'd like to move toward a simpler system where jboss-service.xml isn't involved, default values are set in server.xml (where most other web container config is done) and per-webapp overrides are in jboss-web.xml.

I'd like as little configuration in tc5-cluster-service.xml as possible. Basically, just choosing REPL_SYNC or REPL_ASYNC.

Controlling passivation via configuring eviction policies in tc5-cluster-service.xml runs counter to this. I guess there are global things that have to be configured here, such as the cache loader. (Not being able to persist different webapps' sessions to different stores is actually a flaw in our approach). But most of the other config switches should be configurable per webapp. Having to add separate eviction regions to tc5-cluster-service.xml to do this seems pretty painful.

Also, it seems that once activation/passivation is turned on in tc5-cluster-service.xml, it's going to apply to all webapps, whether needed or not. Is there any way around this, prior to a Region becoming a first-class citizen?

[hmesha]

Brian,

Thanks for your response.

1) Yes it does match my needs to complete the implementation. The only thing is Option was introduced in JBossCache 1.3 and HEAD has 1.2.4SP2 at the moment. I added a new mthod called removeLocal in JBossCacheWrapper that calls TreeCache.evict at the moment ( not to be break the current implemenation) but also has the call for TreeCache.remove(fqn, option) where option = localCacheOverride commented so once HEAD is updated with JBossCache 1.3, I'll switch the call. Also, I changed the remove*Local methods in JBossCacheService to call the new method and not evict. Therefore when the new implementation takes place, the remove*Local will remove sessions from the local cache withot triggering passivation.

2)Shared vs. unshared CacheLoader:

My design is based on unshared cache loader for the same reason you mentioned. If we allow shared cache loader, nodes will step on each other and cause confusion at the application level. I don't see a way to solve this issue other than physically separating the store (unshared CacheLoader) and document it as you mentioned.

3) Notification
3.1) Activation Notification is triggered when TreeCache.get() is called for a session or one of its attributes which is called from
a) CacheListener.nodeModified and nodeRemoved
b) JBossCacheService.getAttribute and loadSession
in either case, post activation event will be dispatched to CacheListener and its nodeActivate method will be executed causing the session to be loaded in the JBossCacheManager. That has been my understanding of what we want the session manager to behave. The rule is "If any of the cluster nodes touch a passivated session and cause the session to be activated in the cache, we would activate it in the session maanger as well". Is this rule not correct? If so, then I can remove the action in CacheListener.nodeActivate method to not load the session and that would solve the issue. The session activation, at this point, would only happen when JBossCacheManager.loadSession() is called on a session whether passivated or already exist in the cache. In other words, we'd decouple the session activation from the cache activation.

3.2) Passivation Notification, that shouldn't be a problem since we only passivate sessions that only exist in JBossCacheManager local store.

4) Configuration

Controlling passivation via configuring eviction policies in tc5-cluster-service.xml runs counter to this. I guess there are global things that have to be configured here, such as the cache loader. (Not being able to persist different webapps' sessions to different stores is actually a flaw in our approach).

I think we can acheive multiple cache loaders by chanining cache loaders but I'd rather leave this as an enhancement in future relases, what you think?

But most of the other config switches should be configurable per webapp. Having to add separate eviction regions to tc5-cluster-service.xml to do this seems pretty painful.

I think, that is the only configuration available at the moment, Any thoughts here?

Also, it seems that once activation/passivation is turned on in tc5-cluster-service.xml, it's going to apply to all webapps, whether needed or not. Is there any way around this, prior to a Region becoming a first-class citizen?

I don't think so. Passivation is trned on or off on a cache loader which is one per TreeCache instance which is shared among all the webapps.
Let me investigate further on this one. will report back

Cheers,

Hany

[brian.stansberry]

Hany, I was looking for ways to make you miserable, so I totally refactored the session repl code over the weekend ;) Sorry about that -- I needed to clean some things up for 4.0.4.CR2 and as I got into it found other issues that required some fairly major surgery to fix. Conceptually most things still work the same though.

"hmesha" wrote:
The rule is "If any of the cluster nodes touch a passivated session and cause the session to be activated in the cache, we would activate it in the session maanger as well". Is this rule not correct?

This is the biggest change we made. Keeping all sessions that are in JBC in JBossCacheManager's session map forces us to do unnecessary deserialization, and thus hurts performance. It also adds complication to the activation/passivation notifications. We now only bring a session into JBossCacheManager if it is accessed on a particular server.

"hmesha" wrote:
If so, then I can remove the action in CacheListener.nodeActivate method to not load the session and that would solve the issue. The session activation, at this point, would only happen when JBossCacheManager.loadSession() is called on a session whether passivated or already exist in the cache. In other words, we'd decouple the session activation from the cache activation.

+1

"hmesha" wrote:
I think we can acheive multiple cache loaders by chanining cache loaders but I'd rather leave this as an enhancement in future relases, what you think?

+1 for deferring this. If chaining cache loaders work, that would be great.

"hmesha" wrote:

But most of the other config switches should be configurable per webapp. Having to add separate eviction regions to tc5-cluster-service.xml to do this seems pretty painful.

I think, that is the only configuration available at the moment, Any thoughts here?

Yes. Don't do eviction via EvictionPolicy; use the Tomcat background thread and do it in backgroundProcess(). Then all the configuration is part of jboss-web.xml. We'd have to add the configurations to the jboss-web.xml DTD of course.

[hmesha]

Hi Brian,

First, let me say that the refactoring that you've done on the session replication is great although made me redo all my changes again :)

Here's what the latest changes refined based on the forum discussion

1. JBossCacheWrapper:

I added 2 methods in there removeLoca(Fqn) and removeLocalSubtree(Fqn) to serve session removal from distributed cache without invoking evict or trigger replication that use the Option class in JBossCache 1.3. Currently, it's uses evict to not break the current implementation until we update head with 1.3 then I'll switch it.

2. JBossCacheService

* I added evictSession method that calls JBossCacheWrapper.evict(Fqn)
* I changed the methods removeAttributeLocal(), removeSessionLocal(), and stop() to call JBossCacheWrapper.removeLocal(Fqn) instead of evict(Fqn)
* I changed the methods removePojoLocal(), and removePojosLocal() to call removeLocalSubtree(Fqn)
* I added a new method isCachePassivationEnabled() that will return ture if the cache loader is not null and the passivation is turned on in the cache loader config.

3. JBossCacheManager

* clearSessions() method is called during the webapp shutdown. It loops through the locally stored sessions to expire them. I invoke isCachePassivationEnabled() method and if it's true, then we passivate the sessions to the cache loader otherwise we let them expire.
* We don't need to do anthing in processExpires() since your refactoring and my changes will allow for removal of sessions that have truly expired (timed out or invalidated) leaving the active sessions to be automatically passivated through the eviction policy (passivation policy) in use on the distrusted store.

4. Cache Loader will always be unshared as we discussed this item on the forum and I'll document that

5. CacheListener

* I changed the class to extend AbstractTreeCacheListener instead of implementing TreeCacheListener to allow me to add the passivate and activate notification methods to it without having to inherit other notification that we're not interested in
* In passivate, If we receive pre-passivation notification from the distrbuted store, we find locally stored session and if it exists we passivate it.
* In activate, we do nothing according to what we agreed about in the last forum post.

6. Configuration:

* I still want to use eviction policy since it's the only way that we can provide the users with the ability to plug their own passivation policy if needed.
* I'm still working on ways to simplify the configuration.

[hmesha]

Hi all,

After a few discussions with Brian over the IM, we have decided to redesign the session passivation based on JBossCache 1.3. Please see the new design on wiki at http://wiki.jboss.org/wiki/Wiki.jsp?page=HttpSessionPassivationDesign

Your feedback is always appreciated.

Cheers,

Hany

[brian.stansberry]

There are 3 caches related to sessions -- the JBC replicated cache, JBossManager.sessions_ and JBossCacheManager.unloadedSessions_. These have to be kept in sync. If a node is evicted from the replicated cache, it needs to also be removed from JBossManager.sessions_. Otherwise, the memory savings from eviction are incomplete. If it is removed from JBossManager.sessions_ an entry needs to be added to JBossCacheManager.unloadedSessions_, with a timestamp equal to the session's lastAccessedTime property. Otherwise the session may never be expired and we'll leak the persistent copy. If it never existed in JBossManager.sessions_, nothing more needs to be done.

Need to be careful about synchronization during this process. .....

[brian.stansberry]

Also, if the session is evicted from the local cache, all nodes that comprise that session should be evicted from JBC.

[brian.stansberry]

A bigger concern is the "maxNodes" configuration. The standard configuration of session passivation is at the session level, and a session will be comprised of one, two or many JBC nodes, depending on granularity. We need to allow configuration of this at the session level; the JBC maxNodes concept is insufficient. Users will want to be able to think in terms of sessions, not the internal implementation detail of nodes.

If FIELD replication is used, I don't know how it is possible to translate from a session config in jboss-web.xml to a maxNodes config for a JBC eviction region. There is no way to know how many nodes will comprise a session.

Also, if FIELD is used, I believe we need to use AopLRUPolicy for it to work correctly.

In general, the translation between session and nodes seems like a can of worms. We get this because we want to use an eviction policy rather than letting the TC background thread do the eviction.