Replication vs. Invalidation and Configuration

[fabrefactor]

Hi JBossCache folks.

We have a number of use cases where we need to maintain state across application servers. Specifically, one application in one cluster may need to save a conversation before redirecting to another application in yet another cluster.

So, the use case is:
* App 1 stores data in the cache
* App 1 redirects to App 2 (via HTTP forward)
* App 2 removes the stored data from the cache

We have had the following problems:
1. We cannot use IP multicast or UDP to let the caches communicate. The only way I have found to get them to discover one another is through the use of TCPPING.
2. It seems as though *sometimes* App 2 cannot find the node in the cache region after redirect, but sometimes it can. The problem is hard to reproduce, as if the App 2 cache joins the App 1 cache for the first time, it works fine, but if the membership was already established, the data is not replicated to App 2. I assume this is because ClusteredCacheLoader fetches the state only on first join.

It would seem as though we need to use synchronous replication rather than invalidation, but the manual seems to advise against that if at all possible.
So I have no idea how to change the configuration to fulfill the use case.

I'll note that we're running under WebSphere 6 and instantiating the conversation cache manually. (Not as a server-wide mbean.)

Thanks in advance for any and all assistance.

Here's our configuration.

<?xml version="1.0" encoding="UTF-8"?>
<server>
 <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar" />
 <!-- ==================================================================== -->
 <!-- Defines TreeCache configuration -->
 <!-- ==================================================================== -->
 <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=TreeCache">
 <depends>jboss:service=Naming</depends>
 <depends>jboss:service=TransactionManager</depends>

 <attribute name="TransactionManagerLookupClass">org.jboss.cache.GenericTransactionManagerLookup</attribute>

 <attribute name="IsolationLevel">REPEATABLE_READ</attribute>

 <attribute name="CacheMode">INVALIDATION_SYNC</attribute>

 <attribute name="UseReplQueue">false</attribute>

 <attribute name="ReplQueueInterval">0</attribute>

 <attribute name="ReplQueueMaxElements">0</attribute>

 <attribute name="ClusterName">onversationCache</attribute>

 <attribute name="ClusterConfig">
 <config>
 <TCP start_port="7800" loopback="true" />
 <TCPPING initial_hosts="localhost[7800],was01[7800],was02[7800],was03[7800],was04[7800],was05[7800],was06[7800],was07[7800],was08[7800]" port_range="20" timeout="3500" num_initial_members="3" up_thread="true" down_thread="true" />
 <MERGE2 min_interval="5000" max_interval="10000" />
 <FD shun="true" timeout="2500" max_tries="5" up_thread="true" down_thread="true" />
 <VERIFY_SUSPECT timeout="1500" down_thread="false" up_thread="false" />
 <pbcast.NAKACK down_thread="true" up_thread="true" gc_lag="100" retransmit_timeout="3000" />
 <pbcast.STABLE desired_avg_gossip="20000" down_thread="false" up_thread="false" />
 <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="false" print_local_addr="true" down_thread="true" up_thread="true" />
 <pbcast.STATE_TRANSFER up_thread="true" down_thread="true" />
 </config>
 </attribute>
 <attribute name="CacheLoaderConfiguration">
 <config>
 <cacheloader>
 <class>org.jboss.cache.loader.ClusteredCacheLoader</class>
 <properties>timeout=3000</properties>
 </cacheloader>
 </config>
 </attribute>

 <attribute name="FetchInMemoryState">true</attribute>

 <attribute name="InitialStateRetrievalTimeout">15000</attribute>

 <attribute name="SyncReplTimeout">15000</attribute>

 <attribute name="LockAcquisitionTimeout">10000</attribute>

 <attribute name="EvictionPolicyClass" />

 <attribute name="UseRegionBasedMarshalling">false</attribute>
 </mbean>
</server>

[fabrefactor]

Anyone, Anyone? Bueller? Bueller?

[fabrefactor]

Hi again, folks.

REPL_SYNC seems to work flawlessly, but we'd prefer not to have to do that.
Is there a way to use only ClusteredCacheLoader and INVALIDATION_SYNC to get data to other nodes in real time?

Thanks in advance.

[manik]

Hi,

Sorry for the slow response.

For your use case where you need to maintain state across all servers, INVALIDATION will not work. You do need to use replication.

Just to brush up on the concepts, invalidation will *remove* state on remote nodes if one node updates this state while replication will *update* this state on all remote nodes.

Invalidation only makes sense if you use a *physical* cache loader (such as a database, file system or far cache) rather than the clustered cache loader, which relies on this state being available on remote cache nodes.

Regarding your multicast issues, I'd recommend troubleshooting your network/JGroups stack first - Here's some literature that may help.

Cheers,
Manik

[fabrefactor]

Thanks for the reply, Manik.

I take it if I'm going to have to use REPL_SYNC that I should also use Buddy Replication (with Data Gravitation) ?

[manik]

I take it if I'm going to have to use REPL_SYNC that I should also use Buddy Replication (with Data Gravitation) ?

That depends on the size of your cluster and the replication overhead you experience.

[fabrefactor]

Thanks very much for your time, Manik.
We'll see how overhead goes without buddy replication.