10 Replies Latest reply on Nov 13, 2007 1:39 PM by Jorge Morales

    JBC 1.4.1SP replication problems

    Jorge Morales Master

      Hi,
      We have a two node cluster configured with replication and a JDBCCacheLoader, as follows:

      <server>
       <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/>
      
       <mbean code="org.jboss.cache.TreeCache"
       name="jboss.cache:service=SOMServicesProvisioningCache">
      
       <depends>jboss:service=Naming</depends>
       <depends>jboss:service=TransactionManager</depends>
       <depends>jboss.jca:name=jdbc/somservices,service=DataSourceBinding</depends>
      
       <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>
       <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
       <attribute name="CacheMode">REPL_SYNC</attribute>
       <attribute name="UseReplQueue">false</attribute>
       <attribute name="ReplQueueInterval">0</attribute>
       <attribute name="ReplQueueMaxElements">0</attribute>
       <attribute name="ClusterName">SOMServicesProvisioning-Cluster</attribute>
      
       <attribute name="ClusterConfig">
      ....
       </attribute>
      
       <attribute name="FetchInMemoryState">true</attribute>
       <attribute name="InitialStateRetrievalTimeout">15000</attribute>
       <attribute name="SyncReplTimeout">15000</attribute>
      
       <attribute name="LockAcquisitionTimeout">10000</attribute>
       <attribute name="EvictionPolicyClass">org.jboss.cache.eviction.LRUPolicy</attribute>
       <attribute name="EvictionPolicyConfig">
       <config>
       <attribute name="wakeUpIntervalSeconds">10</attribute>
       <!-- Cache wide default -->
       <region name="/_default_">
       <attribute name="maxNodes">10000</attribute>
       <attribute name="timeToLiveSeconds">86400</attribute>
       </region>
       </config>
       </attribute>
      
       <attribute name="UseRegionBasedMarshalling">true</attribute>
       <attribute name="CacheLoaderConfiguration">
       <config>
       <passivation>false</passivation>
       <preload>/</preload>
       <shared>true</shared>
       <cacheloader>
       <class>org.jboss.cache.loader.JDBCCacheLoader</class>
       <properties>
       cache.jdbc.datasource=java:jdbc/somservices
       cache.jdbc.table.name=services_provisioning
       cache.jdbc.table.create=true
       cache.jdbc.table.drop=false
       cache.jdbc.table.primarykey=jbosscache_pk
       cache.jdbc.fqn.column=fqn
       cache.jdbc.fqn.type=varchar(255)
       cache.jdbc.node.column=node
       cache.jdbc.node.type=blob
       cache.jdbc.parent.column=parent
       </properties>
       <async>true</async>
       <fetchPersistentState>true</fetchPersistentState>
       <ignoreModifications>false</ignoreModifications>
       </cacheloader>
       </config>
       </attribute>
      
       </mbean>
      </server>
      


      We are having strange behaviour:

      We stop one cluster node.
      We modify one JBC node`s attribute.
      We start the other node of the cluster.


      We expect here to have the correct value, but we have the previously value, I guess that retrieved from Database. We supposed this value was updated when we did a put in the cache, and also that values should be retrieved first from the other node in the cluster, and then, if nothing found, from database.

      Replication seems to be working, because if we programatically stop a region`s synchronization, and put values in that region, when we start that region`s synchronization again, all values are replicated correctly.

      What are we doing wrong?

        • 1. Re: JBC 1.4.1SP replication problems
          Manik Surtani Master

           

          "jorgemoralespou_2" wrote:

          We expect here to have the correct value, but we have the previously value, I guess that retrieved from Database. We supposed this value was updated when we did a put in the cache, and also that values should be retrieved first from the other node in the cluster, and then, if nothing found, from database.


          The cache will first check it's own local memory and after that, check all cache loaders. If you want it to query other caches in the cluster as well, add a ClusteredCacheLoader to your configuration, *before* the JDBCCacheLoader.

          The reason why you may not get the latest value from the JDBCCacheLoader *may* be because you have set this cache loader to be async, so writes may not have completed to the database.

          • 2. Re: JBC 1.4.1SP replication problems
            Jorge Morales Master

             

            The cache will first check it's own local memory and after that, check all cache loaders. If you want it to query other caches in the cluster as well, add a ClusteredCacheLoader to your configuration, *before* the JDBCCacheLoader.

            This seems strange to me, because, we have another cache, with no cache loader configured, and it`s getting it`s state from the other node in the cluster.

            Is this always true, or only when there is another cache loader configured?

            Thanks Manik. Using JBC with your help is always a pleasure.

            • 3. Re: JBC 1.4.1SP replication problems
              Manik Surtani Master

              The ClusteredCacheLoader is not enabled by default.

              Perhaps what you're seeing is simple replication - cache instances in the cluster will replicate state between themselves, and only fall back to using a CacheLoader if the in-memory state has been evicted.

              • 4. Re: JBC 1.4.1SP replication problems
                Jorge Morales Master

                So, to be completely correct, we should have a ClusteredCacheLoader, and then a JBossCacheLoader configured. This will help us with evicted data.
                Normal replication, for puts and removes will always happen, independently of having or not any clustered cache loader.

                Is this right?

                • 5. Re: JBC 1.4.1SP replication problems
                  Manik Surtani Master

                  s/JBossCacheLoader/JDBCCacheLoader/g - then, yes. :-)

                  • 6. Re: JBC 1.4.1SP replication problems
                    Jorge Morales Master

                    Then, for another cache that we don`t persist, but we do evict. Should we configure a ClusteredCacheLoader?
                    Is eviction clustered node dependant? Is eviction in this case, the same as removing the node? Or by evicting the node, and having a ClusteredCacheLoader, we can have data in one node and not in the other, and vice versa, and is only retrieved in case it is searched for? In that case, if we have, for example, an LRUEvictionPolicy, this new data node, retrieved from the other cluster`s node, is put ahead in the eviction queue? Only in this cluster`s node, or in both?

                    I know it`s a handful of questions. Thanks anyway.

                    • 7. Re: JBC 1.4.1SP replication problems
                      Manik Surtani Master

                       

                      "jorgemoralespou_2" wrote:
                      Then, for another cache that we don`t persist, but we do evict. Should we configure a ClusteredCacheLoader?


                      Yes, probably a good idea.

                      "jorgemoralespou_2" wrote:

                      Is eviction clustered node dependant? Is eviction in this case, the same as removing the node? Or by evicting the node, and having a ClusteredCacheLoader, we can have data in one node and not in the other, and vice versa, and is only retrieved in case it is searched for? In that case, if we have, for example, an LRUEvictionPolicy, this new data node, retrieved from the other cluster`s node, is put ahead in the eviction queue? Only in this cluster`s node, or in both?


                      Evictions only happen on a single cache instance and don't care about what's going on in other cache instances in the cluster.

                      • 8. Re: JBC 1.4.1SP replication problems
                        Jorge Morales Master

                        We have a problem with this.
                        Our cache is InactiveOnStartup, and we activate it programatically, so we wait for objects (classes) being fully loaded in the starting cluster node, before getting data nodes replicated from the other cluster node, so we avoid ClassNotFoundExceptions, and such.

                        In the docs, it states that whenever you use an InactiveOnStartup cache, you should use an ExtendedCacheLoader. And ClusteredCacheLoader is not an ExtendedCacheLoader.

                        Is there way to solve this? Probably by a ChainingCacheLoader???

                        Thanks again.

                        • 9. Re: JBC 1.4.1SP replication problems
                          Manik Surtani Master

                          You could create your own class:

                          MyCacheLoader extends ClusteredCacheLoader implements ExtendedCacheLoader
                          { ... }
                          


                          and implement the ExtendedCacheLoader methods as no-ops (or returning a 0-byte array for loadState).

                          ClusteredCacheLoaders should not participate in loading/storing state.

                          • 10. Re: JBC 1.4.1SP replication problems
                            Jorge Morales Master

                            Thanks for youre response. We`ll try it tomorrow.
                            And, yes, youre right, this post is also ours. http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4104272

                            Sorry for crossposting.