Buddy Replication on Weblogic

[vinayakkhamkar]

I have configured two weblogic admin servers on the same machine. A sample application (the one posted on JBoss site, JBoss and Weblogic) using JBossCache is deployed on both these app servers. When Buddy replication is not configured, the contents are replicated. But when buddy replication is enabled at that time, contents are not replicated immediately and there is no specific time interval observed when contents are replicated.

If I access application-1 on admin server and put something into cache and then try to view the cache contents from application-2 on other admin server, they are not reflected there.

Please find the configuration file that I am using. Another thing I observed is I get following error on application startup

===================



===================

2007-04-23 12:11:49,143 ERROR [org.jboss.cache.transaction.DummyTransactionManager] ([ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)') binding of DummyTransactionManager failed
javax.naming.OperationNotSupportedException: bind not allowed in a ReadOnlyContext; remaining name '/TransactionManager'
 at weblogic.jndi.factories.java.ReadOnlyContextWrapper.newOperationNotSupportedException(ReadOnlyContextWrapper.java:145)
 at weblogic.jndi.factories.java.ReadOnlyContextWrapper.newOperationNotSupportedException(ReadOnlyContextWrapper.java:161)
 at weblogic.jndi.factories.java.ReadOnlyContextWrapper.bind(ReadOnlyContextWrapper.java:57)
 at weblogic.jndi.internal.AbstractURLContext.bind(AbstractURLContext.java:45)
 at javax.naming.InitialContext.bind(InitialContext.java:359)
 at org.jboss.cache.transaction.DummyTransactionManager.getInstance(DummyTransactionManager.java:33)
 at org.jboss.cache.DummyTransactionManagerLookup.getTransactionManager(DummyTransactionManagerLookup.java:17)
 at org.jboss.cache.TreeCache._createService(TreeCache.java:1393)
 at org.jboss.cache.TreeCache.createService(TreeCache.java:1378)
 at org.jboss.cache.example.j2eeservices.JBossCacheManager.main(JBossCacheManager.java:74)
 at org.jboss.cache.example.servlet.CacheServlet.init(CacheServlet.java:29)
 at javax.servlet.GenericServlet.init(GenericServlet.java:256)
 at weblogic.servlet.internal.StubSecurityHelper$ServletInitAction.run(StubSecurityHelper.java:276)
 at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
 at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
 at weblogic.servlet.internal.StubSecurityHelper.createServlet(StubSecurityHelper.java:68)
 at weblogic.servlet.internal.StubLifecycleHelper.createOneInstance(StubLifecycleHelper.java:58)
 at weblogic.servlet.internal.StubLifecycleHelper.<init>(StubLifecycleHelper.java:48)
 at weblogic.servlet.internal.ServletStubImpl.prepareServlet(ServletStubImpl.java:504)
 at weblogic.servlet.internal.WebAppServletContext.preloadServlet(WebAppServletContext.java:1698)
 at weblogic.servlet.internal.WebAppServletContext.loadServletsOnStartup(WebAppServletContext.java:1675)
 at weblogic.servlet.internal.WebAppServletContext.preloadResources(WebAppServletContext.java:1595)
 at weblogic.servlet.internal.WebAppServletContext.start(WebAppServletContext.java:2734)
 at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:892)
 at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:336)
 at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:204)
 at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:26)
 at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:60)
 at weblogic.application.internal.flow.ScopedModuleDriver.start(ScopedModuleDriver.java:200)
 at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:117)
 at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:204)
 at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:26)
 at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:60)
 at weblogic.application.internal.flow.StartModulesFlow.activate(StartModulesFlow.java:26)
 at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:641)
 at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:26)
 at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:229)
 at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:154)
 at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:80)
 at weblogic.deploy.internal.targetserver.BasicDeployment.activate(BasicDeployment.java:181)
 at weblogic.deploy.internal.targetserver.BasicDeployment.activateFromServerLifecycle(BasicDeployment.java:352)
 at weblogic.management.deploy.internal.DeploymentAdapter$1.doActivate(DeploymentAdapter.java:52)
 at weblogic.management.deploy.internal.DeploymentAdapter.activate(DeploymentAdapter.java:186)
 at weblogic.management.deploy.internal.AppTransition$2.transitionApp(AppTransition.java:30)
 at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:233)
 at weblogic.management.deploy.internal.ConfiguredDeployments.activate(ConfiguredDeployments.java:169)
 at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:123)
 at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:173)
 at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:89)
 at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
 at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
 at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
2007-04-23 12:11:49,173 DEBUG [org.jboss.cache.TreeCache] ([ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)') Not using an EvictionPolicy

======================

Please find the attached configuration file used for cache configuration of application-1 and application-2.

------------------------------------------------------------------

<?xml version="1.0" encoding="UTF-8"?>

<!-- ===================================================================== -->
<!-- -->
<!-- Sample TreeCache Service Configuration -->
<!-- -->
<!-- ===================================================================== -->

<server>

 <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/>


 <!-- ==================================================================== -->
 <!-- Defines TreeCache configuration -->
 <!-- ==================================================================== -->

 <mbean code="org.jboss.cache.TreeCache"
 name="jboss.cache:service=TreeCache">

 <depends>jboss:service=Naming</depends>
 <depends>jboss:service=TransactionManager</depends>

 <!--
 Configure the TransactionManager
 -->
 <attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute>

 <!--
 Isolation level : SERIALIZABLE
 REPEATABLE_READ (default)
 READ_COMMITTED
 READ_UNCOMMITTED
 NONE
 -->
 <attribute name="IsolationLevel">REPEATABLE_READ</attribute>

 <!--
 Valid modes are LOCAL, REPL_ASYNC and REPL_SYNC
 -->
 <attribute name="CacheMode">REPL_SYNC</attribute>

 <!--
 Just used for async repl: use a replication queue
 -->
 <attribute name="UseReplQueue">false</attribute>

 <!--
 Replication interval for replication queue (in ms)
 -->
 <attribute name="ReplQueueInterval">0</attribute>

 <!--
 Max number of elements which trigger replication
 -->
 <attribute name="ReplQueueMaxElements">0</attribute>

 <!-- Name of cluster. Needs to be the same for all clusters, in order
 to find each other
 -->
 <attribute name="ClusterName">TreeCache-Cluster</attribute>

 <!-- JGroups protocol stack properties. Can also be a URL,
 e.g. file:/home/bela/default.xml
 <attribute name="ClusterProperties"></attribute>
 -->

 <attribute name="ClusterConfig">
 <config>
 <!-- UDP: if you have a multihomed machine,
 set the bind_addr attribute to the appropriate NIC IP address, e.g bind_addr="192.168.0.2"
 -->
 <!-- UDP: On Windows machines, because of the media sense feature
 being broken with multicast (even after disabling media sense)
 set the loopback attribute to true -->
 <UDP mcast_addr="224.10.10.10" mcast_port="45566"
 ip_ttl="64" ip_mcast="true"
 mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
 ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
 loopback="false" bind_addr="127.0.0.1"/>
 <PING timeout="2000" num_initial_members="3"
 up_thread="false" down_thread="false"/>
 <MERGE2 min_interval="10000" max_interval="20000"/>
 <!-- <FD shun="true" up_thread="true" down_thread="true" />-->
 <FD_SOCK/>
 <VERIFY_SUSPECT timeout="1500"
 up_thread="false" down_thread="false"/>
 <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
 max_xmit_size="8192" up_thread="false" down_thread="false"/>
 <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
 down_thread="false"/>
 <pbcast.STABLE desired_avg_gossip="20000"
 up_thread="false" down_thread="false"/>
 <FRAG frag_size="8192"
 down_thread="false" up_thread="false"/>
 <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
 shun="true" print_local_addr="true"/>
 <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
 </config>
 </attribute>


 <!--
 Whether or not to fetch state on joining a cluster
 -->
 <attribute name="FetchStateOnStartup">true</attribute>

 <!--
 The max amount of time (in milliseconds) we wait until the
 initial state (ie. the contents of the cache) are retrieved from
 existing members in a clustered environment
 -->
 <attribute name="InitialStateRetrievalTimeout">15000</attribute>

 <!--
 Number of milliseconds to wait until all responses for a
 synchronous call have been received.
 -->
 <attribute name="SyncReplTimeout">10000</attribute>

 <!-- Max number of milliseconds to wait for a lock acquisition -->
 <attribute name="LockAcquisitionTimeout">15000</attribute>

 <!-- Name of the eviction policy class. Not supported now. -->
 <attribute name="EvictionPolicyClass"></attribute>


<!-- Added for Buddy Replication -->

<!-- Buddy Replication config -->
<attribute name="BuddyReplicationConfig">
 <config>
 <!-- Enables buddy replication. This is the ONLY mandatory configuration element here. -->
 <buddyReplicationEnabled>true</buddyReplicationEnabled>
 <!-- These are the default values anyway -->
 <buddyLocatorClass>org.jboss.cache.buddyreplication.NextMemberBuddyLocator</buddyLocatorClass>
 <!-- numBuddies is the number of backup nodes each node maintains. ignoreColocatedBuddies means that
 each node will *try* to select a buddy on a different physical host. If not able to do so though,
 it will fall back to colocated nodes. -->
 <buddyLocatorProperties>
 numBuddies = 1
 ignoreColocatedBuddies = false
 </buddyLocatorProperties>

 <!-- A way to specify a preferred replication group. If specified, we try and pick a buddy why shares
 the same pool name (falling back to other buddies if not available). This allows the sysdmin to hint at
 backup buddies are picked, so for example, nodes may be hinted topick buddies on a different physical rack
 or power supply for added fault tolerance. -->
 <buddyPoolName>myBuddyPoolReplicationGroup</buddyPoolName>
 <!-- Communication timeout for inter-buddy group organisation messages (such as assigning to and removing
 from groups, defaults to 1000. -->
 <buddyCommunicationTimeout>2000</buddyCommunicationTimeout>
 <!-- Whether data is removed from old owners when gravitated to a new owner. Defaults to true. -->
 <dataGravitationRemoveOnFind>true</dataGravitationRemoveOnFind>
 <!-- Whether backup nodes can respond to data gravitation requests, or only the data owner is supposed to respond.
 defaults to true. -->
 <dataGravitationSearchBackupTrees>true</dataGravitationSearchBackupTrees>
 <!-- Whether all cache misses result in a data gravitation request. Defaults to false, requiring callers to
 enable data gravitation on a per-invocation basis using the Options API. -->
 <autoDataGravitation>false</autoDataGravitation>
 </config>
</attribute>



 <!--
 <attribute name="CacheLoaderClass">org.jboss.cache.loader.bdbje.BdbjeCacheLoader</attribute>
 <attribute name="CacheLoaderConfig">c:\tmp\bdbje</attribute>
 <attribute name="CacheLoaderShared">true</attribute>
 <attribute name="CacheLoaderPreload">/</attribute>
 -->

<!--
 <attribute name="CacheLoaderClass">org.jboss.cache.loader.FileCacheLoader</attribute>
 <attribute name="CacheLoaderConfig">/tmp</attribute>
 <attribute name="CacheLoaderShared">true</attribute>
 <attribute name="CacheLoaderPreload">/</attribute>
-->


 </mbean>


 <!-- Uncomment to get a graphical view of the TreeCache MBean above -->
 <!-- <mbean code="org.jboss.cache.TreeCacheView" name="jboss.cache:service=TreeCacheView">-->
 <!-- <depends>jboss.cache:service=TreeCache</depends>-->
 <!-- <attribute name="CacheService">jboss.cache:service=TreeCache</attribute>-->
 <!-- </mbean>-->


</server>


------------------------------------------------------------------

I also tried to increase "BuddyCommunicationTimeout" but it did not work. Please advice.

[vinayakkhamkar]

I am using JBossCache-1.4.1.SP3 and weblogic9.2 on windows machine.

[genman]

Change

org.jboss.cache.DummyTransactionManagerLookup

to
org.jboss.cache.GenericTransactionManagerLookup

[vinayakkhamkar]

Thanks for the reply, now error related to Transaction lookup is resolved.

But still the contents are not replicated, as in if application-1 puts some value into cache then it is not visible when accessed from application-2.

But when I remove BuddyReplication configuration from the cache-config, everything works fine. Can you please advise on this? or give some more information as in how do we check from logs as how to see if Buddy is found and cache contents are accessed or there was no cache hit?

I have changed the logging to TRACE.

[manik]

But still the contents are not replicated, as in if application-1 puts some value into cache then it is not visible when accessed from application-2.

Buddy Replication requires session affinity. You won't see the contenst on app-2. But it will be on app-2 as a backup so if app-1 crashes you will then be able to see it on app-2.

[vinayakkhamkar]

Manik, Thanks for the reply.

The two nodes are not clustered (sticky session not configured) and I used the same configuration. I have few questions

1) BuddyReplication only works in case of clustered environment? So If I have individual nodes which are not part of cluster then I can't use BuddyReplication?

2) With the same configuration, when I dont use BuddyReplication in my cache-config, still the contents get replicated. i.e Application-1 and Application-2 were able to see the cache contents at any point of time.
(Applications are deployed in two different AdminServers (not a part of cluster) but on single machine). Can you please tell us if that is right and how it replicated the contents when we did not use BuddyReplication and why it is not replicating the contents when we use BuddyReplication?
Is it mandatory to use sticky session for BuddyReplication to work?

[manik]

1) BuddyReplication only works in case of clustered environment? So If I have individual nodes which are not part of cluster then I can't use BuddyReplication?

You don't necessarily need a "cluster" as far as WL is concerned. They could be standalone servers. You will need some form of load balancing and session affinity however, perhaps provided by an external load balancer (HW or SW based)

2) With the same configuration, when I dont use BuddyReplication in my cache-config, still the contents get replicated. i.e Application-1 and Application-2 were able to see the cache contents at any point of time.
(Applications are deployed in two different AdminServers (not a part of cluster) but on single machine). Can you please tell us if that is right and how it replicated the contents when we did not use BuddyReplication and why it is not replicating the contents when we use BuddyReplication?

WL "clustering" has no effect on JBoss Cache, even if you have 2 separate Admin Servers.

Even with BR enabled, replication still happens - it is just that state is replicated to a backup region and not a primary region. This is why you do not see it.

Is it mandatory to use sticky session for BuddyReplication to work?

If it is to be of any benefit, then yes.