5 Replies Latest reply on Mar 8, 2005 4:01 PM by monocongo

    TreeCacheAop Configuration for Synchronous Replication

    monocongo

      I have MBeans which use the TreeCacheAop MBean for storing objects. I want the TreeCacheAop MBeans on all nodes of my cluster to share the same data, i.e. I want to have synchronous replication. At this point it is not working, but I'm hoping that I can configure the TreeCacheAop MBean in such a way that it will.

      As it stands now data is being replicated from the first node to the second node when the second node starts, but after that the replication appears to stop. For example I can start the first node of the cluster and then add two Strings to the TreeCacheAop. I can then start the second node and see the same two Strings in the TreeCacheAop of the second node. However if I add a String to the TreeCacheAop on either node it does not get replicated to the TreeCacheAop on the other node. This is not the behavior I'm trying to achieve, in that if I add data to the TreeCacheAop on one node then I want to see it replicated on the other node, and vice-versa.

      In my SAR I am using the replSync.xml file unaltered from the examples (except that I am using TransactionManagerLookup instead of DummyTransactionManagerLookup), and in the jboss-service.xml file I have the following entry for TreeCacheAop:

       <mbean code="org.jboss.cache.aop.TreeCacheAop"
       name="jboss.cache:service=TreeCacheAop">
       <attribute name="CacheMode">REPL_SYNC</attribute>
       </mbean>
      


      I'm using JBoss 4.0.1 and JBossCache 1.2.

      Thanks in advance for any suggestions on how I might configure things differently in order to get the replication behavior I'm looking for.


      --James

        • 1. Re: TreeCacheAop Configuration for Synchronous Replication

          Are you saying you have two separate xml file to configure the treecacheaop mbean?

          You can turn on the debug tracing (or set it to DEBUG) to see what it actually does.

          -Ben

          • 2. Re: TreeCacheAop Configuration for Synchronous Replication
            monocongo

            Yes I was very confused and was configuring the TreeCache in one deployment descriptor (syncRepl.xml) and the TreeCacheAop in another (jboss-service.xml), since I assumed this was how things worked based on the various documents I've found on this topic (these documents don't seem to be synchronized or complete, my only excuse for making this bonehead mistake). It seemed weird to me at the time but it was sort of working so I thought maybe it was correct, just with the wrong attribute values. Thanks for making your observation which has prompted me to configure the TreeCacheAop MBean in the jboss-service.xml that I include in my SAR.

            Unfortunately having the TreeCacheAop configuration in the right file still doesn't solve my problem, in that replication just isn't happening across the two nodes of my cluster. I currently have the following as my TreeCacheAop MBean entry in my SAR's jboss-service.xml:

             <mbean code="org.jboss.cache.aop.TreeCacheAop"
             name="jboss.cache:service=TreeCacheAop">
             <attribute name="CacheMode">REPL_SYNC</attribute>
             <attribute name="ClusterName">TreeCache-Cluster</attribute>
             <attribute name="ClusterConfig">
             <config>
             <UDP mcast_addr="228.1.2.39"
             mcast_port="45566"
             ip_ttl="64"
             ip_mcast="true"
             bind_addr="localhost"
             mcast_send_buf_size="150000"
             mcast_recv_buf_size="80000"
             ucast_send_buf_size="150000"
             ucast_recv_buf_size="80000"
             loopback="false"/>
             <PING timeout="2000"
             num_initial_members="3"
             up_thread="false"
             down_thread="false"/>
             <MERGE2 min_interval="10000"
             max_interval="20000"/>
             <FD_SOCK/>
             <VERIFY_SUSPECT timeout="1500"
             up_thread="false"
             down_thread="false"/>
             <pbcast.NAKACK gc_lag="50"
             retransmit_timeout="600,1200,2400,4800"
             up_thread="false"
             down_thread="false"/>
             <pbcast.STABLE desired_avg_gossip="20000"
             up_thread="false"
             down_thread="false"/>
             <UNICAST timeout="600,1200,2400"
             window_size="100"
             min_threshold="10"
             down_thread="false"/>
             <FRAG frag_size="8192"
             down_thread="false"
             up_thread="false"/>
             <pbcast.GMS join_timeout="5000"
             join_retry_timeout="2000"
             shun="true"
             print_local_addr="true"/>
             <pbcast.STATE_TRANSFER up_thread="true"
             down_thread="true"/>
             </config>
             </attribute>
             </mbean>
            



            (I don't have the TransactionManagerLookupClass entry because I get exceptions with both DummyTransactionManagerLookup and TransactionManagerLookup, but I'm assuming that's unrelated to the current woes and is something to be addressed later.)

            In server.log I see the following messages which indicate that the TreeCacheAop is being started fine but that it isn't communicating with any other TreeCacheAop in its TreeCache-Cluster, which explains why there is no replication (this is the server.log from the second node, started about 5 minutes after the first node of the cluster, and after there are already data stored in the TreeCacheAop on the first node):

            2005-03-07 15:12:22,033 INFO [org.jboss.cache.aop.TreeCacheAop] interceptor chain is:
            class org.jboss.cache.interceptors.CallInterceptor
            class org.jboss.cache.interceptors.LockInterceptor
            class org.jboss.cache.interceptors.CreateIfNotExistsInterceptor
            class org.jboss.cache.interceptors.ReplicationInterceptor
            
            2005-03-07 15:12:22,033 INFO [org.jboss.cache.aop.TreeCacheAop] cache mode is REPL_SYNC
            
            2005-03-07 15:12:22,054 INFO [STDOUT]
            -------------------------------------------------------
            GMS: address is localhost:37159
            -------------------------------------------------------
            
            2005-03-07 15:12:24,065 INFO [org.jboss.cache.aop.TreeCacheAop]
            viewAccepted(): new members: [localhost:37159]
            
            2005-03-07 15:12:24,066 INFO [org.jboss.cache.aop.TreeCacheAop]
            state could not be retrieved (must be first member in group)
            
            2005-03-07 15:12:24,066 DEBUG [org.jboss.cache.aop.TreeCacheAop]
            Started jboss.cache:service=TreeCacheAop
            
            


            Something else of interest is that when I remove the ClusterConfig attribute from the TreeCacheAop MBean entry I get the following messages instead which show that there is communication between the two TreeCacheAop MBeans, and the current state of the TreeCacheAop on the first node is actually fetched to the newly started TreeCacheAop on the second node, although the replication ends there:

            2005-03-07 17:16:20,299 INFO [org.jboss.cache.aop.TreeCacheAop]
            cache mode is REPL_SYNC
            
            2005-03-07 17:16:20,300 DEBUG [org.jboss.cache.aop.TreeCacheAop]
            setting cluster properties to default value
            
            2005-03-07 17:16:20,323 INFO [STDOUT]
            -------------------------------------------------------
            GMS: address is canal:37219
            -------------------------------------------------------
            
            2005-03-07 17:16:22,340 INFO [org.jboss.cache.aop.TreeCacheAop]
            viewAccepted(): new members: [brooklyn:32989, canal:37219]
            
            2005-03-07 17:16:22,353 INFO [org.jboss.cache.aop.TreeCacheAop]
            received the state (size=818 bytes)
            
            2005-03-07 17:16:22,353 INFO [org.jboss.cache.aop.TreeCacheAop]
            transient state: 766 bytes
            
            2005-03-07 17:16:22,353 INFO [org.jboss.cache.aop.TreeCacheAop]
            setting transient state
            
            2005-03-07 17:16:22,354 DEBUG [org.jboss.cache.lock.IdentityLock]
            Cache instance is null. Use default lock strategy
            
            2005-03-07 17:16:22,354 DEBUG [org.jboss.cache.lock.IdentityLock]
            Cache instance is null. Use default lock strategy
            
            2005-03-07 17:16:22,355 DEBUG [org.jboss.cache.lock.IdentityLock]
            Cache instance is null. Use default lock strategy
            
            2005-03-07 17:16:22,364 INFO [org.jboss.cache.aop.TreeCacheAop]
            locking the old tree
            
            2005-03-07 17:16:22,364 INFO [org.jboss.cache.aop.TreeCacheAop]
            locking the old tree was successful
            
            2005-03-07 17:16:22,364 INFO [org.jboss.cache.aop.TreeCacheAop]
            setting the transient state was successful
            
            2005-03-07 17:16:22,365 INFO [org.jboss.cache.aop.TreeCacheAop]
            forcing release of all locks in old tree
            
            2005-03-07 17:16:22,365 INFO [org.jboss.cache.aop.TreeCacheAop]
            state was retrieved successfully (in 25 milliseconds
            
            2005-03-07 17:16:22,366 DEBUG [org.jboss.cache.aop.TreeCacheAop]
            Started jboss.cache:service=TreeCacheAop
            


            Notice above that when the ClusterConfig is left out then the actual name of the server, canal, is listed in the GMS info message, whereas before it was listed as localhost. Does this tell us anything ?

            Thanks for any more advice or suggestions.


            --James

            • 3. Re: TreeCacheAop Configuration for Synchronous Replication
              monocongo

              I have not included the setting for transaction support in my TreeCacheAop configuration because I have been getting errors when using the TransactionManagerLookupClass attribute, with either DummyTransactionManagerLookup or TransactionManagerLookup as this attribute's value. I now expect that this absence of transaction support is why I do not get replication with the TreeCacheAop.


              When I use the TransactionManagerLookup class for this attribute's value


               <attribute name="TransactionManagerLookupClass">org.jboss.cache.TransactionManagerLookup</attribute>
              



              I get the following Exception:


              09:28:28,845 ERROR [TreeCacheAop] Starting failed jboss.cache:service=TreeCacheAop
              
              java.lang.InstantiationException: org.jboss.cache.TransactionManagerLookup
              
               at java.lang.Class.newInstance0(Class.java:335)
               at java.lang.Class.newInstance(Class.java:303)
               at org.jboss.cache.TreeCache.startService(TreeCache.java:1089)
               at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:272)
               at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:222)
               at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
               at java.lang.reflect.Method.invoke(Method.java:585)
               at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:144)
               at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
               at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
               at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
               at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:642)
               at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:891)
               at $Proxy0.start(Unknown Source)
              
               etc...
              




              When I use the DummyTransactionManagerLookup class (the original setting in the example syncRepl.xml) for this attribute's value


               <attribute name="TransactionManagerLookupClass">org.jboss.cache.TransactionManagerLookup</attribute>
              



              I get the following Exception:


              09:37:22,322 ERROR [DummyTransactionManager] binding of DummyTransactionManager failed
              
              javax.naming.NameAlreadyBoundException
              
               at org.jnp.server.NamingServer.bind(NamingServer.java:126)
               at org.jnp.interfaces.NamingContext.bind(NamingContext.java:501)
               at org.jnp.interfaces.NamingContext.bind(NamingContext.java:469)
               at javax.naming.InitialContext.bind(InitialContext.java:359)
               at org.jboss.cache.transaction.DummyTransactionManager.getInstance(DummyTransactionManager.java:41)
               at org.jboss.cache.DummyTransactionManagerLookup.getTransactionManager(DummyTransactionManagerLookup.java:17)
               at org.jboss.cache.TreeCache.startService(TreeCache.java:1094)
               at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:272)
               at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:222)
               at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
               at java.lang.reflect.Method.invoke(Method.java:585)
               at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:144)
               at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
               at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
               at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
               at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:642)
               at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:891)
               at $Proxy0.start(Unknown Source)
               at org.jboss.system.ServiceController.start(ServiceController.java:416)
              
               etc...
              



              One item of interest is that the TreeCacheAop still deploys OK when this error happens, in contrast to the failure to deploy which happens as a result of using the TransactionManagerLookup class for this attribute's value.

              Any ideas on what causes these errors and/or how to get around them ? Thanks in advance for any suggestions.


              --James


              • 4. Re: TreeCacheAop Configuration for Synchronous Replication

                You should JBossTransactionManagerLookup for the attribute.

                -Ben

                • 5. Re: TreeCacheAop Configuration for Synchronous Replication
                  monocongo

                  Thanks for this information. This takes care of the transaction support, but unfortunately the replication is still not working as advertised.

                  I will go back over to the various documents I've found on JBossCache and see if there is anything else I may have missed or done wrong. If any of the above deployment descriptor entries look fishy then I'd really appreciate hearing about it.

                  Thanks in advance, and for all of the help so far.


                  --James