Work on JBCACHE-13: Standalone Tomcat http session replicati

[bstansberry]

For the last few weeks I've been working on getting JBossCache-based session replication to work in standalone Tomcat 5.5. I've gotten a pretty reasonable version working, and want to post what I doing on this thread so others can be informed and provide any inputs they would like. (I've been discussing things with Ben Wang and earlier with Remy Maucherant).

Basically this task breaks down into two main elements -- getting a TreeCache deployed when Tomcat starts and getting JBossCacheManager instances deployed when distributable webapps are deployed.

To do this, I've:

1) Created a new implementation of the TC Cluster interface,
JBossCacheCluster. It's role is basically to:

a) Interact with the TC deployment system -- get config attributes from server.xml, implement the TC Lifecycle interface.
b) Instantiate a TreeCache and register it in JMX.
c) Instantiate and configure JBossCacheManager instances
when requested by the Tomcat deployer.

2) Created new class StandaloneJBossCacheManager, subclass of JBossCacheManager. I expect that with a bit of refactoring this subclass' functionality can be merged back into JBossCacheManager. Instances of this class can be created by JBossCacheCluster, or directly by TC's webapp deployer code. The latter happens if the user specifies a element in the context.xml the describes their webapp. This latter approach allows a user to set the replicationGranularity and replicationTrigger atttributes to something other than the cluster-wide defaults. In embedded JBoss/Tomcat these two attributes are set via jboss-web.xml.

Added functionality in StandaloneJBossCacheManager is basically:

a) Exposes a few property attributes so JBossCacheCluster or the TC deployer can set them without using the WebMetaData object used with embedded JBoss/Tomcat.
b) Handles the start()/stop() process differently, to account for the fact that it can't count on being configured by JBoss' TomcatDeployer. Key things it must do:

-- Detect if it was instantiated by the TC deployer rather than JBossCacheCluster. If it was, it finds the JBossCacheCluster and passes itself to it so the cluster can set cluster-wide properties (e.g. snapshot mode, useJK, etc).
-- Instantiate and install its ClusteredSessionValve.


I'll discuss some of the issues related to the above in separate posts.

[bstansberry]

Farmed Deployment and the Tomcat Cluster interface

To do the JBCACHE-13 work we are creating a new implementation of Tomcat's Cluster interface. Tomcat's existing Cluster implementation not only provides session replication but also supports farmed deployments of webapps. The primary issue of this post is to discussion whether an/or how we want to support farm deployments.

Tomcat's existing implementation of its Cluster interface is the class org.apache.catalina.cluster.SimpleTcpCluster. This class actually provides two main services -- HttpSession replication across the cluster and a form of farm deployment of webapps across the cluster. The Cluster interface itself clearly specifies the requirements for session replication. It also includes some methods related to farm deployments, but these methods are all deprecated and are actually only implemented as no-ops by SimpleTcpCluster. SimpleTcpCluster actually implements farm deployment using a separate subinterface, CatalinaCluster.

We need to decide whether we want to support a farm deployment capability in the new Cluster implementation. The methods related to farm deployment are deprecated in the interface and are not really implemented by the existing implementation. But, the existing implementation does support farm deployments, so not doing it in the new implementation would represent a loss of functionality.

IF we choose to provide farmed deployment support, there are a number of approaches to getting it done:

1) Have the new JBossCacheCluster class subclass SimpleTcpCluster and inherit its farm deployment functionality. The concern I have with this approach is it's really bolting one semi-complex class on top of another complex class, with the two classes being maintained by separate, unrelated people.
2) Have JBossCacheCluster itself instantiate and deploy an instance of SimpleTcpCluster and delegate farm deployment work to it. This would be similar to what it already does with its TreeCache instance. This would loosen the coupling with the SimpleTcpCluster code, but there would still be the danger of incompatible changes being made in SimpleTcpCluster. I need to look further to see if this is even feasible.
3) Basically reimplement SimpleTcpCluster's farm deployment functionality inside JBossCacheCluster. An issue here is SimpleTcpCluster uses TC's own cluster messaging code (they can't use JGroups 'cause of LGPL). It seems odd for a JBoss project to not use JGroups.
4) Come up with a new farm deployment implementation using JGroups and/or JBossCache. This would be a big increase in the scope of the project.

Any thoughts on this would be most appreciated :)

[bstansberry]

Cluster Configuration

To deploy a JBossCacheCluster, the user would add an element like the following to the Host section of TC's server.xml file, replacing the standard Cluster element:

<Cluster
 className="org.jboss.web.tomcat.tc5.JBossCacheCluster"
 cacheObjectName="jboss.cache:service=TomcatClusteringCache"
 clusterName="Tomcat-Cluster"
 clusterProperties="conf/jgroups-default.xml"
 isolationLevel="REPEATABLE_READ"
 cacheMode="REPL_ASYNC"
 lockAcquisitionTimeout="15000"
 snapshotMode="instant"
 snapshotInterval="2000"
 useJK="false"
 defaultReplicationTrigger="SET_AND_NON_PRIMITIVE_GET"
 defaultReplicationGranularity="attribute"
 useLocalCache="true"\
transactionManagerLookupClass="org.jboss.cache.UserTransactionTransactionManagerLookup"
/>

Most of the attributes have the exact same function as the equivalent attributes in the JBoss -service.xml files. I'll provide reasonable defaults for them, so most could be ignored. A few deserve comment:

1) clusterProperties This attribute controls configuration of the underlying JGroups channel. The problem here is the most effective way to configure this is with an embedded XML element (the way its handled in the JBoss MBean deployment descriptors). But, as far as I can tell the Commons Digester library that TC uses to parse server.xml does not support parsing text as an Element and passing the resulting Element to an object as a property value. (Someone please tell me if I'm wrong).

This makes configuring JGroups more difficult, and probably the easiest way to do it is with an external config file.

The clusterProperties attribute can take one of three kinds of values:

a) A relative path to a JGroups config file located under $CATALINA_BASE. If such a file can be found, JBossCacheCluster will convert it to a file URL and pass it to JGroups, which parses it.
b) A URL pointing to a JGroups config file.
c) An old-style JGroups configuration string.

2) cacheObjectName Object name under which the TreeCache is to be registered. This serves two purposes:

a) Before instantiating a TreeCache, JBossCacheCluster looks to see if a cache is already registered under the given cacheObjectName. If one is, that cache is used (and any other cache configuration attributes are ignored).
b) If no cache is already registered under cacheObjectName, JBossCacheCluster creates the TreeCache and registers it under that name.

This approach gives users the flexibility to configure different caches under different Host elements, but also to have separate Cluster elements under different Hosts share a TreeCache.

Currently Tomcat only supports nesting a Cluster element under a Host element. It seems logical to me that this element also be allowed under the Engine element. I'll take this up on the Tomcat dev list. (I got this working with no trouble using JBossCacheCluster; haven't had a chance to see if SimpleTcpCluster will work as a child of Engine).

3) transactionManagerLookupClass Has the same function as it does in the standard TreeCache deployment descriptors. The tricky thing here is the user will need to configure a TransactionManger and have Tomcat register it in JNDI. I've done this using the GlobalJndiResources element in server.xml, which I believe is the standard way.

For my dev work I've used JOTM as my transaction manager. Will JBossJTA run outside of JBoss?

I've written a new implementation of TransactionManagerLookup that looks for a transaction manager in JNDI under "java:/UserTransaction". With JOTM, a single class implements both UserTransaction and TransactionManager, so typically a user registering JOTM would register it under java:/UserTransaction so webapps could conveniently get a UserTransaction.

[starksm64]

I don't see why session replication and farming should be combined in the same interface. They should be two seperately configured aspects of clustering.

We have not done any work to run the JBossJTA service outside of 4.0.x. A minimal 5.x microcontainer should allow for use of the jta service, but nothing to test this has been done as yet so far as I know.

[bstansberry]

Quick note on what's going on with this:

Code basically along the lines discussed above has been checked into Branch_4_0 and HEAD.

JBossCacheCluster has been changed in that it does not expose attributes to configure the TreeCache in detail (e.g. "isolationLevel", "cacheMode", etc.). Instead an attribute is exposed where the user can configure a relative path to a jboss-service.xml-style MBean deployment descriptor that configures the cache. If the attribute is not set, JBossCacheCluster will look for a file named cache-cluster.xml in the Tomcat conf folder. If no file is found, the TreeCache will be configured using reasonable defaults.

No need to worry about a TransactionManager; DummyTransactionManager works fine. Using JOTM would be a mistake, as that would potentially involve session replication with user transactions.

All of StandaloneJBossCacheManager's functionality has been merged into JBossCacheManager. JBossCacheManager can determine based on how it's configured whether its running embedded or standalone.

JBossCacheManager handles deploying its ClusteredSessionValve itself; this has been removed from TomcatDeployer. The manager needs to be able to do this in standalone mode, so I saw no point in having the code duplicated.

[bstansberry]

Testing

I would definitely want to add some unit tests to the JBoss testsuite for standalone session replication. I have tested it manually, but automated testing would be far superior. I believe it should be possible to get many of the existing unit tests to run in standalone mode as well.

Here's what I'm thinking. Any comments from testsuite gurus or Tomcat gurus (or anyone else) would be much appreciated:

1) Testsuite build adds a target that builds up a minimal Tomcat installation in the testsuite/output folder. Most of the required pieces are already in the thirdparty module for use in the tomcat sar; would need to bring over a few more. Installation should include Tomcat's Manager webapp.

2) Testsuite adds a target(s) that is able to create a directory for use as a $CATALINA_BASE and programatically launch a Tomcat instance configured to use it.

3) Two such instances would be launched.

4) "Standalone" versions of the existing session replication test cases would be created. These would use the same test code, but instead of using the standard JBoss mechanism for deploying their war files into JBoss, a new version would be created that deploys the war files using HTTP calls to Tomcat's Manager webapp.


If this seems like overkill or there is a simple way, please let me know :)

[starksm64]

I would view this a new macro that launches a tomcat dist vs the current jboss dist which is what the testsuite/imports/server-config.xml start-jboss macro does.

#1 The issue is how to integrate the standalone footprint into the testsuite. This could conditionally done based on a catalina.home property that would point to an external dist.

#2 Since we have to install jboss specific jars or at least make them available to the tomcat instance classpath, one could also argue that this is really no different than then existing tomcat-ssl-tests, tomcat-sso-clustered-tests, etc. where a minimal jboss mc is used to host the tomcat instance.

Which approach to use is a function of where we want people to download tomcat and follow a recipe to add jboss clustering, or whether a customized tomcat install/installer is used.

[bstansberry]

I would view this a new macro that launches a tomcat dist vs the current jboss dist which is what the testsuite/imports/server-config.xml start-jboss macro does.

Yes, that's exactly what I was thinking.

#1 The issue is how to integrate the standalone footprint into the testsuite. This could conditionally done based on a catalina.home property that would point to an external dist.

Using an external dist is definitely more straightforward than trying to create one in the testsuite. As you said, the issue is getting the required JBoss jars on the server classpath. The following jars need to be in $CATALINA_HOME/server/lib; getting them there is certainly simple enough if we don't mind copying them into the external dist.

tomcat55-service.jar (or a stripped down version thereof)
jboss-cache.jar
jgroups.jar
jboss.jar
jboss-j2ee.jar
jboss-system.jar
jboss-jmx.jar
jboss-common.jar
concurrent.jar
log4j.jar

There would need to be two catalina.base directory structures to hold the server.xml files, logs, temp and work dirs for the two instances. Two distinct server.xml's would be needed, configured to use different ports, as I don't believe tomcat supports an analogue to the jboss.bind.addr feature.

Since we have to install jboss specific jars or at least make them available to the tomcat instance classpath, one could also argue that this is really no different than then existing tomcat-ssl-tests, tomcat-sso-clustered-tests, etc. where a minimal jboss mc is used to host the tomcat instance.

A concern I have about using a minimal jboss mc as the test platform is that we need to be sure that webapps are deployed using the standalone tomcat deployment code, not TomcatDeployer. Making sure the deployment works right is really the main thing to test. Once an app is deployed, the actual mechanics of session replication are exactly the same in standalone mode vs. embedded. (There are classloader hierarchy differences that make standalone replication a different use case that needs to be tested, but the code being executed is the same).

Which approach to use is a function of where we want people to download tomcat and follow a recipe to add jboss clustering, or whether a customized tomcat install/installer is used.

I'd been thinking in terms of a recipe approach, as it is fairly simple and allows users to easily integrate jboss clustering into different Tomcat revs. (The dependencies of jboss clustering on TC code is very minimal, so the integration is unlikely to break from rev to rev). A rough outline of a distribution zip might be as follows.

doc
--- .... various docs
dist
--- conf
------ server-cache.xml
------ cluster-cache.xml
--- server
------ lib
--------- .... the libs listed above
src
--- main
------ ... relevant source from the jboss-tomcat module
build.xml

The recipe would be to:
1) Copy the contents of the dist folder to your Tomcat distribution.
2) Either rename server-cache.xml to server.xml, or cut and paste the relevant pieces to your existing server.xml
3) Edit cluster-cache.xml if you want to change the TreeCache config.
4) If you want to override the default replication granularity or trigger for a particular webapp, add a element to its context.xml.

If we want to distribute source, I wouldn't distribute all the source from the tomcat module; only the portions relevant to JBossCache based session and sso replication. The other classes in the module introduce compile-time dependencies on the cluster, connector and security modules.

Ben and I discussed not including JBossCache and its dependencies in a session replication distribution and having them integrate the current JBossCache distribution. Ben, since we talked I've realized that JBossCache distributes the jboss-minimal.jar while the session repl code needs the full jboss.jar. Given this difference, I think having users integrate would be pretty tough.

Sorry if this is wordy :)

[jkharness87]

First, I aplogize if this is an inappropriate forum for this question. I searched the forums and felt this topic was most appropriate for this forum since it was the only one I could find specifically related to standalone tomcat integaration with JBossCache. Kindly redirect me, if this is not the correct place.

I have been attempting to follow the instructions contained within this forum topic to cluster the HttpSession of a standalone tomcat instance. Specifically, I am using JBossCache 1.3.0SP1 (and the distributed libraries that come with it), Tomcat 5.5.16 on Windows XP and tomcat55-service.jar from JBoss 4.0.4 CR2.

Here is a snippet from my tomcat server.xml file:

 <Cluster
 className="org.jboss.web.tomcat.tc5.session.JBossCacheCluster"
 cacheObjectName="jboss.cache:service=TomcatClusteringCache"
 clusterName="Tomcat-Cluster"
 isolationLevel="REPEATABLE_READ"
 cacheMode="REPL_SYNC"
 lockAcquisitionTimeout="15000"
 snapshotMode="instant"
 snapshotInterval="2000"
 useJK="false"
 defaultReplicationTrigger="SET_AND_NON_PRIMITIVE_GET"
 defaultReplicationGranularity="attribute"
 useLocalCache="false"
 transactionManagerLookupClass="org.jboss.cache.DummyTransactionManage
rLookup"
 />

Tomcat (misleadingly, IMO) outputs the following upon startup:

2006-05-01 13:33:21,225 ERROR [JBossCacheService] (main) jboss.cache:service=Tom
catClusteringCache service to Tomcat clustering not found
2006-05-01 13:33:21,225 ERROR [ContainerBase] (main) ContainerBase.setManager: s
tart:
LifecycleException: jboss.cache:service=TomcatClusteringCache service to Tomcat
 clustering not found: org.jboss.web.tomcat.tc5.session.ClusteringNotSupportedE
xception: jboss.cache:service=TomcatClusteringCache service to Tomcat clustering
 not found
 at org.jboss.web.tomcat.tc5.session.JBossCacheManager.startUnembedded(JB
ossCacheManager.java:1207)
 at org.jboss.web.tomcat.tc5.session.JBossCacheManager.start(JBossCacheMa
nager.java:355)
 at org.apache.catalina.core.ContainerBase.setManager(ContainerBase.java:
431)
 at org.apache.catalina.core.StandardContext.start(StandardContext.java:4
124)
 at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase
.java:759)
 at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:73
9)
 at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:524)

 at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:809)

 at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:698
)
 at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:472
)
 at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1112)
 at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java
:310)
 at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(Lifecycl
eSupport.java:119)
 at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1021)

 at org.apache.catalina.core.StandardHost.start(StandardHost.java:718)
 at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1013)

 at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442
)
 at org.apache.catalina.core.StandardService.start(StandardService.java:4
50)
 at org.apache.catalina.core.StandardServer.start(StandardServer.java:709
)
 at org.apache.catalina.startup.Catalina.start(Catalina.java:551)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:275)
 at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

Stepping through the code with the eclipse debugger (using the jpda remote debugger), reveals that the JBossCacheService constructor catches a ClassCastException. It is thrown from the following line:

proxy_ = (TreeCacheAop) ((TreeCacheAopMBean) MBeanProxyExt.create(TreeCacheAopMBean.class, cacheServiceName_)).getInstance();

It looks like the code is expecting a TreeCacheAopMBean to be registered with JMX. However, if I understand things correctly, a TreeCacheMBean is registered in JMX. Thus, the getInstance() method returns an instance of TreeCache. As a result casting it to a TreeCacheAop fails.

I think I have analyzed this correctly, although I could be wrong. Could someone please point me in the right direction? It appears as if this was working at one point in time. I'm extremley interested in getting this up and running so I can test it.

[brian.stansberry]

Very appropriate forum :)

The work on JBCLUSTER-13 stopped due to priorities and so the code in JBossCacheCluster fell out of sync w/ the rest of the session repl code, which now uses TreeCacheAop.

I just committed a change to JBossCacheCluster on Branch_4_0 that may fix your problem; you can get it from CVS.

Thanks for the report.

[jkharness87]

Brian -

Thanks for the quick fix. I checked out the source and that fixed the problem. Another quick question:

It seems like the cache is using the jboss jmx server implementation (please correct me if I am wrong). Is there a way to configure it to use tomcat's embedded jmx server or is that a codebase change?

Many Thanks,

Josh Harness

Senior Engineer
Jewelry Television

[jkharness87]

Brian -

Thanks for the quick fix. I checked out the source and that fixed the problem. Another quick question:

It seems like the cache is using the jboss jmx server implementation (please correct me if I am wrong). Is there a way to configure it to use tomcat's embedded jmx server or is that a codebase change?

Many Thanks,

Josh Harness

Senior Engineer
Jewelry Television

[brian.stansberry]

The cache is registered in JMX by JBossCacheCluster; it uses the Tomcat JMX server.

[jkharness87]

Brian -

A few more questions:

1. I tried pre-compiling with aopc and it looks like the checkPojoType method of org.jboss.web.tomcat.tc5.session is not correctly detecting that my POJO implements the Advised interface. Does it work for you with aopc?

2. I was able to get it to succesfully work with the -javagent command line switch for Java 5 (using loadtime weaving). However, I'm not happy with a couple of things and hoped that you could shed some light:

A. I had to modify the catalina.sh/catalina.bat script such that gobs of tomcat specific jars that are in $CATALINA_HOME/common/lib and $CATALINA_HOME/server/lib (to avoid ClassNotFoundException's). Am I doing something wrong? Seems these should already be in the classpath.

B. I'd like it if my POJO classes that are being clustered don't have to reside in $CATALINA_HOME/common/lib. It seems that they have to reside there because if they are deployed as a part of a War file, when the second tomcat instance starts up, it attempts to receive the clustered objects from tomcat 1's session. However, since this occurs prior to loading my distributable war in tomcat 2, a ClassNotFoundException is thrown. Do you know of any way around this?

3. Is there any way, that you know of, through the aop notation in jboss-aop.xml to instruct jboss-aop to instrument all field access for a package name that starts with, for instance, com.jtv no matter how deep the package structure goes? I'd like to not have to declare com.jtv.bo.*, com.jtv.services.*, etc. Honestly, I'd really like it if everything I placed in the HttpSession auto-magically @ loadtime was forced to implement the Advised interface and had field access interception applied. Of course, I realize this probably is just a dream ; )

4. Ultimatley, we'd like to potentially have multiple war files deployed that use different revisions of our clusted POJO's in tomcat. We'd want to do this, for instance, when we are draining user sessions off of a web context that we are upgrading with a new web context (with updated objects that should be clustered). How would we configure this such that the new war doesn't join the wrong cluster given that it seems that the cluster is configured on a tomcat-wide basis rather than on a web context by web context basis? Is this possbile, that you know of, with JBossCache and tomcat?

FYI - I'm currently @ Java One and just noticed you are speaking along with Ben Wang @ a BOF session. I greatly look forward to meeting you guys and hearing the insights you have.

Many Thanks,

Josh Harness

Senior Engineer
Jewelry Televison

[kabirkhan]

"jkharness87" wrote:
Brian -
3. Is there any way, that you know of, through the aop notation in jboss-aop.xml to instruct jboss-aop to instrument all field access for a package name that starts with, for instance, com.jtv no matter how deep the package structure goes? I'd like to not have to declare com.jtv.bo.*, com.jtv.services.*, etc.

You can use a wildcard in your pointcut expression, this will match zero or more characters. A few examples

field(java.lang.String org.acme.*->test)
field(java.lang.String org.acme.*->*)
field(* org.acme.*->*)

The first will match all fields called "test" of type String in any class within any subpackage of org.acme (so it would match org.acme.Blah and org.acme.sub.package.SomeClass)

The second will follow the same class/package rules but match all fields of type String regardless of the name.

The third will match all fields of all classes under org.acme regardless of their name or type