Clustering configuration for GateIn-Tomcat packaging

Version 5

    This article provides instructions on establishing a cluster system for GateIn-Tomcat packaging, we aim at building two nodes connecting to the same MySQL datasources. In the scope of this article, we are only interested in replication in service layer, there will not be configuration for HTTP session replication (that Tomcat internally supports) and there is no need to create a load balancer component.

     

    1. Prepare GateIn-Tomcat binaries

     

    We create directory gatein-cluster designed to contain all relevant resources . Then, copy GateIn-Tomcat packaging directory to its two subdirectories tomcat_0 and tomcat_1

     

    cd gatein-cluster
    
    cp -rf {GateIn_Project_Dir}/packaging/tomcat/tomcat7/target/tomcat  tomcat_0
    cp -rf {GateIn_Project_Dir} /packaging/tomcat/tomcat7/target/tomcat tomcat_1
    
    

     

    NOTES: There was a switch from JBossCache to Infinispan for PicketLink caching in recent releases which led to JGroups conflicts, GateIn 3.4.x or earlier releases are recommended (GateIn 3.4.0-Final was used in this tutorial). Unlike JBoss AS7, Tomcat does not offer a fine-grained class loading mechanism to resolve that JGroups conflict.

     

    2. Database configuration

     

    2.1. Prepare MySQL schemas

     

    Create two schemas named jdbcjcr and jdbcidm and grant all privileged on those schemas to user 'gatein' (whose password is 123456)

     

    2.2. Datasource parameters in configuration.properties

     

    Adapt datasource parameters in tomcat_0/gatein/conf/configuration.properties and tomcat_1/gatein/conf/configuration.properties to MySQL

     

     

    gatein.jcr.datasource.driver=com.mysql.jdbc.Driver
    gatein.jcr.datasource.url=jdbc:mysql://localhost:3306/jdbcjcr
    gatein.jcr.datasource.username=gatein
    gatein.jcr.datasource.password=123456
    
    gatein.idm.datasource.driver=com.mysql.jdbc.Driver
    gatein.idm.datasource.url=jdbc:mysql://localhost:3306/jdbcidm
    gatein.idm.datasource.username=gatein
    gatein.idm.datasource.password=123456
    
    

     

     

    2.3. JDBC Driver

     

    Copy mysql jdbc connector to tomcat_0/lib and tomcat_1/lib, then get rid of hsqldb.jar as there is no longer need of it.

     

    3. Shared directory

     

    For search consistency, it is critical that both cluster nodes operate on a common Lucene index. A simple way to do so is to redirect Lucene index storage to a shared directory.

     

    Let's create directory shared_dir under gatein-cluster, then assgin relative path to this directory to the parameter gatein.data.dir in gatein/conf/configuration.properties under both tomcat_0 and tomcat_1

     

     

    gatein.data.dir=../../shared_dir
    

     


    4. JCR custom config

     

    Turn value of parameter gatein.jcr.index.changefilterclass from

     

     

    gatein.jcr.index.changefilterclass=org.exoplatform.services.jcr.impl.core.query.DefaultChangesFilter
    

     

    to

     

    gatein.jcr.index.changefilterclass=org.exoplatform.services.jcr.impl.core.query.jbosscache.JBossCacheIndexChangesFilter
    

     

     

    5. Resolve port conflict

     

    To avoid port conflict in case the two cluster nodes are running on a single machine, we need to assure that ports of Tomcat connectors and debugger agents from two nodes take different values. One practical way is to keep the port numbers unchanged on tomcat_0 and increase any port number from tomcat_1 by 1000.

     

     

    6. JVM properties

     

    Remaining work is to provide following JVM properties in tomcat_0/bin/gatein-dev.sh and tomcat_1/bin/gatein-dev.sh

     

     

    -Dexo.profiles=cluster -Djava.net.preferIPv4Stack=true
    

     

    1. -Dexo.profiles=cluster  specifies that service and cache components in Portal Container are initialized in cluster mode
    2. -Djava.net.preferIPv4Stack=true fixes welknown IPv6 issue with JGroups

     

    7. Establish test scenario

     

      We deploy a service component that facilitate the test of cace replication (keep in mind that our cluster nodes are connecting to the same database server)

     

    Checkout the code source from GitHub  https://github.com/mto/cluster-test-module , then build it and copy the artifact to tomcat_0/lib and tomcat_1/lib

     

    NOTE: The version of exo.kernel on GitHub repo is 2.3.6-GA, users should adjust it to kernel version used in underlying GateIn.

     

     

    7.1. ClusterTestService

     

     

    package org.gatein.cluster.testservice;
    
    import org.exoplatform.management.annotations.Impact;
    import org.exoplatform.management.annotations.ImpactType;
    import org.exoplatform.management.annotations.Managed;
    import org.exoplatform.management.annotations.ManagedDescription;
    import org.exoplatform.management.annotations.ManagedName;
    import org.exoplatform.management.jmx.annotations.NameTemplate;
    import org.exoplatform.management.jmx.annotations.Property;
    import org.exoplatform.management.rest.annotations.RESTEndpoint;
    import org.exoplatform.services.cache.CacheService;
    import org.exoplatform.services.cache.ExoCache;
    import org.picocontainer.Startable;
    
    /**
     * @author <a href="hoang281283@gmail.com">Minh Hoang TO</a>
     * @date 4/4/13
     */
    @Managed
    @NameTemplate({@Property(key = "view", value = "portal"), @Property(key = "service", value = "clustertest"),
       @Property(key = "type", value = "clustertest")})
    @ManagedDescription("ClusterTestService")
    @RESTEndpoint(path = "clustertestservice")
    public class ClusterTestService implements Startable{
    
        private ExoCache<String, String> cache;
    
        public ClusterTestService(CacheService cacheService)
        {
            cache = cacheService.getCacheInstance(ClusterTestService.class.getSimpleName());
        }
    
        @Managed
        @ManagedDescription("Add element to underlying cache")
        @Impact(ImpactType.WRITE)
        public void addElement(@ManagedDescription("key") @ManagedName("key") String key, @ManagedDescription("value") @ManagedName("value") String value)
        {
            cache.put(key, value);
        }
    
        @Managed
        @ManagedDescription("Show elements available on underlying cache")
        @Impact(ImpactType.READ)
        public String showElements() throws Exception
        {
            StringBuilder b = new StringBuilder();
            for(String item : cache.getCachedObjects())
            {
                b.append(" " + item);
            }
            return b.toString();
        }
    
        @Override
        public void start() {
        }
    
        @Override
        public void stop() {
        }
    }
    
    

     

     

       Instance of ClusterTestService holds a cache that is replicated while running in cluster mode. The class exposes two methods via JMX/REST

    that enables user to add new element into the cache and to show all elements available on cache.

     

    7.2. Service and cache configuration

     

    Configuration for ClusterTestService and associating distributed cache is declared in  conf/portal/configuration.xml as following

     

     

    <configuration
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.exoplaform.org/xml/ns/kernel_1_2.xsd http://www.exoplaform.org/xml/ns/kernel_1_2.xsd"
       xmlns="http://www.exoplaform.org/xml/ns/kernel_1_2.xsd">
    
      <component>
        <key>org.gatein.cluster.testservice.ClusterTestService</key>
        <type>org.gatein.cluster.testservice.ClusterTestService</type>
      </component>
    
      <external-component-plugins>
        <target-component>org.exoplatform.services.cache.CacheService</target-component>
        <component-plugin>
              <name>addExoCacheConfig</name>
              <set-method>addExoCacheConfig</set-method>
              <type>org.exoplatform.services.cache.ExoCacheConfigPlugin</type>
              <description>add Exo Cache Config</description>
              <init-params>
                <object-param>
                  <name>cache.config.ClusterTestService</name>
                  <description>The JBoss Cache configuration for the cluster test service</description>
                  <object type="org.exoplatform.services.cache.ExoCacheConfig">
                    <field name="name">
                      <string>ClusterTestService</string>
                    </field>
                    <field name="maxSize">
                      <int>${gatein.cache.description.maxsize:5000}</int>
                    </field>
                    <field name="liveTime">
                      <long>${gatein.cache.description.livetime:600}</long>
                    </field>
                    <field name="implementation">
                      <string>org.exoplatform.services.cache.concurrent.ConcurrentFIFOExoCache</string>
                    </field>
                  </object>
                </object-param>
              </init-params>
        </component-plugin>
        <component-plugin profiles="cluster">
          <name>addExoCacheConfig</name>
          <set-method>addExoCacheConfig</set-method>
          <type>org.exoplatform.services.cache.ExoCacheConfigPlugin</type>
          <description>add Exo Cache Config</description>
          <init-params>
            <object-param>
              <name>cache.config.ClusterTestService</name>
              <description>The JBoss Cache configuration for the cluster test service</description>
              <object type="org.exoplatform.services.cache.impl.jboss.ea.EAExoCacheConfig">
                <field name="name">
                  <string>ClusterTestService</string>
                </field>
                <field name="expirationTimeout">
                  <long>${gatein.cache.description.expiration:600000}</long>
                </field>
                <field name="maxNodes">
                 <int>${gatein.cache.description.maxnodes:5000}</int>
                </field>
                <field name="distributed">
                  <boolean>true</boolean>
                </field>
              </object>
            </object-param>
          </init-params>
        </component-plugin>
      </external-component-plugins>
    
    </configuration>
    
    

     

    8. Test cache replication

     

    Start two cluster nodes with the commands

     

    gatein-cluster/tomcat_0/bin/gatein-dev.sh run
    
    gatein-cluster/tomcat_1/bin/gatein-dev.sh run
    

     

    Start JConsole and  establish connections to both Tomcat instances. Open MBeans tab on each connection view and browse to clustertestservice JMX bean

     

    1. Invoke method addElement("user", "gatein") from one JMX connection view
    2. Invoke method showElements() from the other JMX connection view to see if "gatein" appears in the result

     

    9. Related links

     

    Other resources on GateIn clustering

     

    https://docs.jboss.org/author/display/GTNPORTAL35/Clustering+configuration

    https://community.jboss.org/wiki/GateInClustering