Session affinity, large cluster, use of invalidation, shared cache loader

IMPORTANT NOTE:

This page discusses the relevant concepts in terms of "HTTP Session replication".  The configuration below is not a supported configuration for http session replication in the JBoss Application Server, and will not necessarily work properly if implemented. Readers of this page should think of the concepts on this page in terms of how to manage cached data that is logically "owned" by one server in a cluster, but which needs to be available to other servers in case of a failover. An HttpSession is conceptually that kind of data, but meeting the servlet spec and the full set of expected behaviors of an HttpSession mean that a sophisticated integration layer is required on top of the JBoss Cache.  The current integration layer in JBoss AS may not work reliably with the configuration below.

Author

Manik Surtani (manik@jboss.org)

Use case

You have a large cluster of application servers and the overhead of replicating state is high.  Network traffic and memory usage on each node is a problem and scaling is difficult.  Data may be accessed on different nodes, although this is infrequent and typically sessions are sticky.

Prerequisites

To use this pattern, you must:

• Use a load balancer

• Use sticky sessions

• JBoss Cache 1.3.0 "Wasabi" and above

Description

JBoss Cache 1.3.0 has several features that can be used to help this scenario, in this case:

• Invalidation

• Chaining cache loaders

Since sessions are sticky, there is often no need for each node in the cluster to have access to all session data.  We are concerned though that this data may live on another server as well and hence cannot rely on suppressing replication as this will result in stale data.  Here is where we use a new cache mode, Invalidation.  Invalidation offers you a chance to send evict messages to the cluster when data changes.  evict messages are small and efficient, and will ensure that other caches in the cluster mark data as stale if the data has been modified elsewhere.

We revert back to a cache loader again to deal with the case of retrieving data and picking up from a failed node.  A far cache may be used, but in this case I will consider a shared JDBC cache loader.

Keep in mind that in JBoss Cache 1.3.0 onwards, cache loaders can even be chained.

Sample configuration

Configuring JBoss Cache on your cluster

<?xml version="1.0" encoding="UTF-8"?>

<server>

    <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"></classpath>

    <mbean code="org.jboss.cache.TreeCache"
        name="jboss.cache:service=TreeCache">

        <depends>jboss:service=Naming</depends>
        <depends>jboss:service=TransactionManager</depends>

        <attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute>

        <attribute name="IsolationLevel">REPEATABLE_READ</attribute>

        <attribute name="CacheMode">INVALIDATION_ASYNC</attribute>

        <attribute name="UseReplQueue">true</attribute>

        <attribute name="ReplQueueInterval">2000</attribute>

        <attribute name="ReplQueueMaxElements">25</attribute>

        <attribute name="ClusterName">TreeCache-Cluster</attribute>

        <attribute name="ClusterConfig">
            <config>
                <UDP mcast_addr="228.1.2.3" mcast_port="48866"
                    ip_ttl="64" ip_mcast="true" 
                    mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
                    ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
                    loopback="false"></UDP>
                <PING timeout="2000" num_initial_members="3"
                    up_thread="false" down_thread="false"></PING>
                <MERGE2 min_interval="10000" max_interval="20000"></MERGE2>
                <FD_SOCK></FD_SOCK>
                <VERIFY_SUSPECT timeout="1500"
                    up_thread="false" down_thread="false"></VERIFY_SUSPECT>
                <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
                    max_xmit_size="8192" up_thread="false" down_thread="false"></pbcast>
                <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
                    down_thread="false"></UNICAST>
                <pbcast.STABLE desired_avg_gossip="20000"
                    up_thread="false" down_thread="false"></pbcast>
                <FRAG frag_size="8192"
                    down_thread="false" up_thread="false"></FRAG>
                <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
                    shun="true" print_local_addr="true"></pbcast>
                <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"></pbcast>
            </config>
        </attribute>

        <attribute name="LockAcquisitionTimeout">10000</attribute>

        <attribute name="UseMarshalling">false</attribute>

        <attribute name="CacheLoaderConfiguration">
            <config>
                <passivation>false</passivation>
                <shared>true</shared>

                <cacheloader>
                    <class>org.jboss.cache.loader.JDBCCacheLoader</class>
                    <properties>
                        cache.jdbc.table.name=jbosscache
                        cache.jdbc.table.create=true
                        cache.jdbc.table.drop=true
                        cache.jdbc.table.primarykey=jbosscache_pk
                        cache.jdbc.fqn.column=fqn
                        cache.jdbc.fqn.type=varchar(255)
                        cache.jdbc.node.column=node
                        cache.jdbc.node.type=longblob
                        cache.jdbc.parent.column=parent
                        cache.jdbc.driver=com.mysql.jdbc.Driver
                        cache.jdbc.url=jdbc:mysql://db_host.mycorp.com:3306/jbossdb
                        cache.jdbc.user=root
                        cache.jdbc.password=
                    </properties>
                    <async>true</async>
                    <fetchPersistentState>false</fetchPersistentState>
                    <ignoreModifications>true</ignoreModifications>
                </cacheloader>
            </config>
        </attribute>

    </mbean>
</server>

Referenced by: