Shared TreeCache for Session Replication and SSO

[bstansberry]

This thread disappeared from the forum last week -- I'm about to post a new message, so I thought I'd reconstruct the thread....

On 5/25/04 Brian Stansberry wrote:

A while ago I added package o.j.web.tomcat.tc5.sso which uses a TreeCache to provide cross-cluster support for Tomcat's Single Sign On valve. Scott suggested that it would be good if the sso code could share a TreeCache with the JBossCacheManager that will be used for session replication.


I'm writing to be sure I'm on the same page with those of you working on the JBossCacheManager.


I've tweaked the sso code so it exposes an MBean attribute for its TreeCache, which at a minimum would allow the Tomcat5 MBean to set the tree cache to some shared object. However, I saw in the jboss-service-50.xml file in HEAD a reference to a shared tree-cache mbean "jboss.cache:service=TreeCache". This led me to a couple of questions:


1) Is the use of "jboss.cache:service=TreeCache" still the plan? If so, I'll look at updating the sso code to try to find this tree cache service as its default behavior, only creating its own cache if it can't find the shared service.


2) What is the planned configuration of "jboss.cache:service=TreeCache"? The tree cache used by the sso is configured for REPL_SYNC, with READ_COMMITTED transaction isolation.

On 5/25/04 Ben Wang replied

Hi,


The JBossCacheManager code was checked in by Bela. I plan to take over and finish it next month. As for the service name, yes, service=TreeCache has been a naming convention. So mostly likely, we will stick with it.


My guess is for state replication, REPL_SYNC and READ_COMMITTED will be used as well.


BTW, has your code been checked in yet?


-Ben

On 5/26 Brian Stansberry replied:

Yes, it's in 3.2 and HEAD, although I need to port a small change to the HEAD version. I plan on making some changes tonight to have it look for the service=TreeCache cache.


FYI, I designed the sso assuming an unshared cache, and had been storing data in the cache in the following structure:


/ABC12..../credentials --- credential data needed to support client authentication on another node for sso id ABC12...
/ABC12.../sessions --- set of all the HttpSession id's associated with sso ABC12....
/XYZ98...../credentials
/XYZ98...../sessions


Ids for sso's are basically analogous to HttpSession ids.


Since this data will now be in a shared cache, would you want all of the above under some sort of higher level structure, e.g. /sso/ABC12..../credentials ? If so, will that have performance implications due to locking -- e.g. while the /sso/XYZ98.../sessions node is being written, the /sso/ABC12.../credentials node is locked for read? Sorry, my knowledge of TreeCache locking is still limited :(


The credentials and sessions were stored in separate nodes rather than as attributes of a single node because the sso manager needs to be able to respond to a rare situation where the credential node is updated after it is created. I implement that using TreeCacheListener.nodeUpdated(), and in that method ignore the many changes to the "sessions" node.


Brian

[bstansberry]

In then end having the clustered sso valve try to find a shared TreeCache but then create its own if it could not find the shared one led to a lot of complexity. Having the valve manage the lifecycle of a created cache is not clean at all. IMO it seems much cleaner to simply externalize the TreeCache and have Tomcat5 pass in the object name of the tree cache service.

I've changed the ClusteredSingleSignOn to work this way (although the code to create the TreeCache is just disabled, and can be reenabled fairly easily if someone disagrees with this approach).

I've added a "cacheName" attribute to Tomcat5, and use the value of this attribute to configure any ClusteredSingleSignOn valve. The default value of the attribute is "jboss.cache:service=TreeCache".

My next question is where do we want to configure the jboss.cache:service=TreeCache service? For testing I did this in the Tomcat 5 sar's jboss-service.xml, but I'm not sure that's where you want the config to go. The webserver config file seems a non-intuitive place to put a service with such a generic name. But if we do it elsewhere, we need to have a note in the Tomcat sar config file saying where.

[belaban]

Hi Brian,

yes, the right way is to pass the name of a TreeCache MBean, which you can configure externally, e.g. in a different XML file.

Bela

[bstansberry]

Hi Bela,

Has there been any decision about where the TreeCache MBean config file will be? If the purpose of the tree cache is just for handling Tomcat-related replication, having the tree cache mbean config in the same jboss-service.xml as the tomcat service makes sense. If the tree cache is for all kinds of use, obviously it needs to be somewhere else.

Brian

[belaban]

The XML file can be anywhere, e.g. tree-service.xml in ./deploy.

But I suggest to ship jbossweb-tomcat5.sar/META-INF/jboss-service.xml with its own TreeCache configuration (e.g. for session repl), and have SSO refer to it via ObjectName.
People can always comment it if they don't use it.
Bela

[starksm64]

Ultimately I would think that there is a tree cache attribut associated with the current cluster service that replaces the DistributedState service. I don't see a need in general to have a seperate configuration of the jgroups stack for every clustering usecase. Can tree cache run on top of the current cluster-service.xml stack today?

[belaban]

Ultimately I would think that there is a tree cache attribut associated with the current cluster service that replaces the DistributedState service. I don't see a need in general to have a seperate configuration of the jgroups stack for every clustering usecase. Can tree cache run on top of the current cluster-service.xml stack today?

No, it can't. Currently it runs directly on top of JGroups. Reason is that this makes it easier for a standalone distribution.
Bela

[bstansberry]

bela wrote:

But I suggest to ship jbossweb-tomcat5.sar/META-INF/jboss-service.xml with its own TreeCache configuration (e.g. for session repl), and have SSO refer to it via ObjectName.

If you and Scott decide to have the config in jbossweb-tomcat5.sar/META-INF/jboss-service.xml, we need to give the service a less generic name. I suggest, jboss.cache:service=TreeCache,type=jboss.web

BTW, from painful experience I learned the more obvious "jboss.web:service=TreeCache" is no good. The Tomcat5 service's stop() method unregisters all MBeans under the jboss.web space (it assumes they are all Tomcat objects). So, when the ServiceController goes to stop the TreeCache service, it is no longer registered in the JMX server and an exception is thrown.

[belaban]

Hey Stan,

you're in Fremont, want to meet some time to discuss things ? I'm in San Jose CA.
I think having the TreeCache MBean reside in Tomcat's XML file is not such a good idea, because we'd be creating a TreeCache even if *none* of the webapps are marked as .
So we should just *refer* to an existing TreeCache MBean, not embed one.
Note that I'm working on exposing the TreeCache via JNDI as well.
Bela

[starksm64]

His name is actually Brian Stansberry, not Stan. I agree that the tree cache service config should not be in the jboss-service.xml of the tomcat sar. Ultimately this should replace the existing jbossha-httpsession.sar used for session replication when both sso and session replication can sit on top of the tree cache.

[bstansberry]

Hi Bela,

Earlier I'd replied off the forum, but don't know if the address I had was still valid.

Anyway, sure, it would be great to get together to compare notes. Will you be at the svjug meeting on 6/9?

Best,
Brian -- aka Stan ;-)

[bonhamcm]

So did Bela and Brian get together to discuss the solution to this problem? I noticed JBoss 3.2.4 final was released without an example (although setting up a separate TreeCache service was pretty easy).

[bonhamcm]

We upgraded to JBoss 3.2.4 final and there's no longer an example for the ClusteredSingleSignOnValve (There was in JBoss 3.2.4RC2).
So, I added this to server.xml:

<Valve className="org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn"
debug="0"
treeCacheName="jboss.cache:service=TreeCache,type=jboss.web"/>

And this to the Tomcat5 jboss-service.xml file:

<mbean code="org.jboss.cache.TreeCache"
name="jboss.cache:service=TreeCache,type=jboss.web">

<attribute name="ClusterName">DefaultPartition</attribute>
<attribute name="ClusterProperties">sso-channel.xml</attribute>
<attribute name="CacheMode">REPL_SYNC</attribute>
<attribute name="IsolationLevel">READ_COMMITTED</attribute>
<attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>

</mbean>

<mbean code="org.jboss.web.tomcat.tc5.Tomcat5"
name="jboss.web:service=WebServer">

Unfortunately, it looks like the authentication credentials are lost again. Hopefully this is a configuration issue. I did notice this lines in the log:

18:08:55,769 ERROR [GroupRequest] both corr and transport are null, cannot send group request
18:08:55,769 ERROR [GroupRequest] both corr and transport are null, cannot send group request

Any insights?

[bstansberry]

The configuration you wrote looks good to me. I tested it on my home network and it worked.

The one suggestion I have is checking the configuration of the UDP protocol in the sso-channel.xml file. If one or more of the servers in your cluster run on a multihomed machine, you should uncomment the protocol config version that has a "bind_addr" attribute and set it to the IP address you want. e.g.

<config>
 <!-- UDP: if you have a multihomed machine, comment out the first protocol version,
 uncomment the second and set the bind_addr attribute to the appropriate NIC IP address -->
 <!-- UDP: On Windows machines, because of the media sense feature
 being broken with multicast (even after disabling media sense)
 set the loopback attribute to true -->

 <!--
 <UDP mcast_recv_buf_size="800000" mcast_send_buf_size="800000" mcast_port="45567"
 ucast_recv_buf_size="800000" mcast_addr="228.8.8.8" loopback="false"
 ucast_send_buf_size="800000" ip_ttl="32"/>
 -->

 <UDP bind_addr="192.100.100.202" mcast_recv_buf_size="800000" mcast_send_buf_size="800000" mcast_port="45567"
 ucast_recv_buf_size="800000" mcast_addr="228.8.8.8" loopback="false"
 ucast_send_buf_size="800000" ip_ttl="32"/>

 <PING timeout="2000" .....

Sorry for the slow reply. Before replying I wanted to test your config, but my network has been down :(