This article provides instructions on establishing a cluster system for GateIn-Tomcat packaging, we aim at building two nodes connecting to the same MySQL datasources. In the scope of this article, we are only interested in replication in service layer, there will not be configuration for HTTP session replication (that Tomcat internally supports) and there is no need to create a load balancer component.
1. Prepare GateIn-Tomcat binaries
We create directory gatein-cluster designed to contain all relevant resources . Then, copy GateIn-Tomcat packaging directory to its two subdirectories tomcat_0 and tomcat_1
cd gatein-cluster
cp -rf {GateIn_Project_Dir}/packaging/tomcat/tomcat7/target/tomcat tomcat_0
cp -rf {GateIn_Project_Dir} /packaging/tomcat/tomcat7/target/tomcat tomcat_1
NOTES: There was a switch from JBossCache to Infinispan for PicketLink caching in recent releases which led to JGroups conflicts, GateIn 3.4.x or earlier releases are recommended (GateIn 3.4.0-Final was used in this tutorial). Unlike JBoss AS7, Tomcat does not offer a fine-grained class loading mechanism to resolve that JGroups conflict.
2. Database configuration
2.1. Prepare MySQL schemas
Create two schemas named jdbcjcr and jdbcidm and grant all privileged on those schemas to user 'gatein' (whose password is 123456)
2.2. Datasource parameters in configuration.properties
Adapt datasource parameters in tomcat_0/gatein/conf/configuration.properties and tomcat_1/gatein/conf/configuration.properties to MySQL
gatein.jcr.datasource.driver=com.mysql.jdbc.Driver gatein.jcr.datasource.url=jdbc:mysql://localhost:3306/jdbcjcr gatein.jcr.datasource.username=gatein gatein.jcr.datasource.password=123456 gatein.idm.datasource.driver=com.mysql.jdbc.Driver gatein.idm.datasource.url=jdbc:mysql://localhost:3306/jdbcidm gatein.idm.datasource.username=gatein gatein.idm.datasource.password=123456
2.3. JDBC Driver
Copy mysql jdbc connector to tomcat_0/lib and tomcat_1/lib, then get rid of hsqldb.jar as there is no longer need of it.
3. Shared directory
For search consistency, it is critical that both cluster nodes operate on a common Lucene index. A simple way to do so is to redirect Lucene index storage to a shared directory.
Let's create directory shared_dir under gatein-cluster, then assgin relative path to this directory to the parameter gatein.data.dir in gatein/conf/configuration.properties under both tomcat_0 and tomcat_1
gatein.data.dir=../../shared_dir
4. JCR custom config
Turn value of parameter gatein.jcr.index.changefilterclass from
gatein.jcr.index.changefilterclass=org.exoplatform.services.jcr.impl.core.query.DefaultChangesFilter
to
gatein.jcr.index.changefilterclass=org.exoplatform.services.jcr.impl.core.query.jbosscache.JBossCacheIndexChangesFilter
5. Resolve port conflict
To avoid port conflict in case the two cluster nodes are running on a single machine, we need to assure that ports of Tomcat connectors and debugger agents from two nodes take different values. One practical way is to keep the port numbers unchanged on tomcat_0 and increase any port number from tomcat_1 by 1000.
6. JVM properties
Remaining work is to provide following JVM properties in tomcat_0/bin/gatein-dev.sh and tomcat_1/bin/gatein-dev.sh
-Dexo.profiles=cluster -Djava.net.preferIPv4Stack=true
- -Dexo.profiles=cluster specifies that service and cache components in Portal Container are initialized in cluster mode
- -Djava.net.preferIPv4Stack=true fixes welknown IPv6 issue with JGroups
7. Establish test scenario
We deploy a service component that facilitate the test of cace replication (keep in mind that our cluster nodes are connecting to the same database server)
Checkout the code source from GitHub https://github.com/mto/cluster-test-module , then build it and copy the artifact to tomcat_0/lib and tomcat_1/lib
NOTE: The version of exo.kernel on GitHub repo is 2.3.6-GA, users should adjust it to kernel version used in underlying GateIn.
7.1. ClusterTestService
package org.gatein.cluster.testservice;
import org.exoplatform.management.annotations.Impact;
import org.exoplatform.management.annotations.ImpactType;
import org.exoplatform.management.annotations.Managed;
import org.exoplatform.management.annotations.ManagedDescription;
import org.exoplatform.management.annotations.ManagedName;
import org.exoplatform.management.jmx.annotations.NameTemplate;
import org.exoplatform.management.jmx.annotations.Property;
import org.exoplatform.management.rest.annotations.RESTEndpoint;
import org.exoplatform.services.cache.CacheService;
import org.exoplatform.services.cache.ExoCache;
import org.picocontainer.Startable;
/**
* @author <a href="hoang281283@gmail.com">Minh Hoang TO</a>
* @date 4/4/13
*/
@Managed
@NameTemplate({@Property(key = "view", value = "portal"), @Property(key = "service", value = "clustertest"),
@Property(key = "type", value = "clustertest")})
@ManagedDescription("ClusterTestService")
@RESTEndpoint(path = "clustertestservice")
public class ClusterTestService implements Startable{
private ExoCache<String, String> cache;
public ClusterTestService(CacheService cacheService)
{
cache = cacheService.getCacheInstance(ClusterTestService.class.getSimpleName());
}
@Managed
@ManagedDescription("Add element to underlying cache")
@Impact(ImpactType.WRITE)
public void addElement(@ManagedDescription("key") @ManagedName("key") String key, @ManagedDescription("value") @ManagedName("value") String value)
{
cache.put(key, value);
}
@Managed
@ManagedDescription("Show elements available on underlying cache")
@Impact(ImpactType.READ)
public String showElements() throws Exception
{
StringBuilder b = new StringBuilder();
for(String item : cache.getCachedObjects())
{
b.append(" " + item);
}
return b.toString();
}
@Override
public void start() {
}
@Override
public void stop() {
}
}
Instance of ClusterTestService holds a cache that is replicated while running in cluster mode. The class exposes two methods via JMX/REST
that enables user to add new element into the cache and to show all elements available on cache.
7.2. Service and cache configuration
Configuration for ClusterTestService and associating distributed cache is declared in conf/portal/configuration.xml as following
<configuration
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.exoplaform.org/xml/ns/kernel_1_2.xsd http://www.exoplaform.org/xml/ns/kernel_1_2.xsd"
xmlns="http://www.exoplaform.org/xml/ns/kernel_1_2.xsd">
<component>
<key>org.gatein.cluster.testservice.ClusterTestService</key>
<type>org.gatein.cluster.testservice.ClusterTestService</type>
</component>
<external-component-plugins>
<target-component>org.exoplatform.services.cache.CacheService</target-component>
<component-plugin>
<name>addExoCacheConfig</name>
<set-method>addExoCacheConfig</set-method>
<type>org.exoplatform.services.cache.ExoCacheConfigPlugin</type>
<description>add Exo Cache Config</description>
<init-params>
<object-param>
<name>cache.config.ClusterTestService</name>
<description>The JBoss Cache configuration for the cluster test service</description>
<object type="org.exoplatform.services.cache.ExoCacheConfig">
<field name="name">
<string>ClusterTestService</string>
</field>
<field name="maxSize">
<int>${gatein.cache.description.maxsize:5000}</int>
</field>
<field name="liveTime">
<long>${gatein.cache.description.livetime:600}</long>
</field>
<field name="implementation">
<string>org.exoplatform.services.cache.concurrent.ConcurrentFIFOExoCache</string>
</field>
</object>
</object-param>
</init-params>
</component-plugin>
<component-plugin profiles="cluster">
<name>addExoCacheConfig</name>
<set-method>addExoCacheConfig</set-method>
<type>org.exoplatform.services.cache.ExoCacheConfigPlugin</type>
<description>add Exo Cache Config</description>
<init-params>
<object-param>
<name>cache.config.ClusterTestService</name>
<description>The JBoss Cache configuration for the cluster test service</description>
<object type="org.exoplatform.services.cache.impl.jboss.ea.EAExoCacheConfig">
<field name="name">
<string>ClusterTestService</string>
</field>
<field name="expirationTimeout">
<long>${gatein.cache.description.expiration:600000}</long>
</field>
<field name="maxNodes">
<int>${gatein.cache.description.maxnodes:5000}</int>
</field>
<field name="distributed">
<boolean>true</boolean>
</field>
</object>
</object-param>
</init-params>
</component-plugin>
</external-component-plugins>
</configuration>
8. Test cache replication
Start two cluster nodes with the commands
gatein-cluster/tomcat_0/bin/gatein-dev.sh run gatein-cluster/tomcat_1/bin/gatein-dev.sh run
Start JConsole and establish connections to both Tomcat instances. Open MBeans tab on each connection view and browse to clustertestservice JMX bean
- Invoke method addElement("user", "gatein") from one JMX connection view
- Invoke method showElements() from the other JMX connection view to see if "gatein" appears in the result
9. Related links
Other resources on GateIn clustering
https://docs.jboss.org/author/display/GTNPORTAL35/Clustering+configuration
Comments