13 Replies Latest reply on Jun 29, 2006 3:31 AM by ben.wang

    Replicated field values not 'sticking' in clustered POJO Cac

    mbenner

      I'm rather stumped by this problem. I've implemented POJO Cache (1.3.0 SP2) standalone within Resin app server.
      I have two servers (ServerA and ServerB) within the same subnet that each start a TreeCacheAOP instance upon server startup and add a POJO to the TreeCache as such:

      *****

      // Configure and start the TreeCache
      cache = new TreeCacheAop();
      
      PropertyConfigurator config = new PropertyConfigurator();
      config.configure(cache, "./WEB-INF/replSync-service.xml");
      
      cache.start();
      
      MerchantDailyAccount mda = new MerchantDailyAccount();
      mda.setDailySpend(128.00F);
      
      cache.putObject("/test/dailySpend", mda);
      

      *******

      Let's say that ServerA starts up first and adds a MerchantDailyAccount object to the cache (as shown above) with a 'dailySpend' value of 128.00.
      When ServerB starts up, it also adds a MerchantDailyAccount object to the cache with the same 'dailySpend' value (128.00) and replicates the object across back the network to ServerA (I see it in the logs).

      The problem is that when I inspect the cached MerchantDailyAccount objects on ServerA and ServerB, I see the following:


      ServerA: MerchantDailyAccount.dailySpend = 1.00 (its initial value defined within the POJO)
      ServerB: MerchantDailyAccount.dailySpend = 128.00 (the correct value)


      Somehow, the 'dailySpend' value on ServerA has gotten overwritten and reverted to its initial state.
      Here's where it gets even stranger. If I then retrieve the cached MerchantDailyAccount object from ServerA and increment
      its value by 0.50, I will see the following:


      ServerA: MerchantDailyAccount.dailySpend = 128.50 (the correct value)
      ServerB: MerchantDailyAccount.dailySpend = 128.00 (should see 128.50, as on ServerA)


      In the logs on ServerB, I see the replication transaction being sent from ServerA attempting to update the value of 'dailySpend' to 128.50 on ServerB.

      I've poured over the logs on both servers and don't see any errors that would indicate that the replication event failed (in fact there are no
      ERROR messages recorded in the logs on either server). I see the correct values being passed back and forth between the two servers, but it
      just doesn't look like the values 'stick' in the server being replicated to. Its almost like the transaction isn't being effectively committed until I access and change the value in the object replicated to.

      Has anyone seen this before or have any idea what might be causing this. I've included my XML configuration below, in case its helpful. Thank you
      for your help!

      Michael


      <?xml version="1.0" encoding="UTF-8"?>
      
      <server>
      
       <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/>
      
       <mbean code="org.jboss.cache.TreeCache"
       name="jboss.cache:service=TreeCache">
      
       <depends>jboss:service=Naming</depends>
       <depends>jboss:service=TransactionManager</depends>
      
       <attribute name="TransactionManagerLookupClass">org.jboss.cache.GenericTransactionManagerLookup</attribute>
      
       <attribute name="NodeLockingScheme">PESSIMISTIC</attribute>
      
       <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
      
       <attribute name="CacheMode">REPL_SYNC</attribute>
      
       <attribute name="UseReplQueue">false</attribute>
      
       <attribute name="ReplQueueInterval">0</attribute>
      
       <attribute name="ReplQueueMaxElements">0</attribute>
      
       <attribute name="ClusterName">GiftsCom-TreeCache-Cluster</attribute>
      
       <attribute name="ClusterConfig">
       <config>
       <UDP mcast_addr="228.1.2.3" mcast_port="48866"
       ip_ttl="64" ip_mcast="true"
       mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
       ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
       loopback="false"/>
       <PING timeout="2000" num_initial_members="3"
       up_thread="false" down_thread="false"/>
       <MERGE2 min_interval="10000" max_interval="20000"/>
       <FD_SOCK/>
       <VERIFY_SUSPECT timeout="1500"
       up_thread="false" down_thread="false"/>
       <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
       max_xmit_size="8192" up_thread="false" down_thread="false"/>
       <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
       down_thread="false"/>
       <pbcast.STABLE desired_avg_gossip="20000"
       up_thread="false" down_thread="false"/>
       <FRAG frag_size="8192"
       down_thread="false" up_thread="false"/>
       <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
       shun="true" print_local_addr="true"/>
       <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
       </config>
       </attribute>
      
       <attribute name="FetchInMemoryState">false</attribute>
      
       <attribute name="InitialStateRetrievalTimeout">20000</attribute>
      
       <attribute name="SyncReplTimeout">20000</attribute>
      
       <attribute name="LockAcquisitionTimeout">15000</attribute>
      
       <attribute name="EvictionPolicyClass">org.jboss.cache.eviction.AopLRUPolicy</attribute>
      
       <attribute name="EvictionPolicyConfig">
       <config>
       <attribute name="wakeUpIntervalSeconds">5</attribute>
       <region name="/_default_">
       <attribute name="maxNodes">5000</attribute>
       <attribute name="timeToLiveSeconds">1000</attribute>
       </region>
       <region name="/test/">
       <attribute name="maxNodes">10000</attribute>
       <attribute name="timeToLiveSeconds">5</attribute>
       </region>
       </config>
       </attribute>
      
       <attribute name="UseMarshalling">true</attribute>
      
       </mbean>
      
      </server>


        • 1. Re: Replicated field values not 'sticking' in clustered POJO
          manik

          Are you sure you are reading your config file properly?

          PropertyConfigurator config = new PropertyConfigurator();
          config.configure(cache, "./WEB-INF/replSync-service.xml");
          


          will look in your classpath for ./WEB-INF/replSync-service.xml. I don't think this is what you want.

          Try putting the xml file in WEB-INF/classes and use:

          PropertyConfigurator config = new PropertyConfigurator();
          config.configure(cache, "replSync-service.xml");
          


          That ought to work.


          • 2. Re: Replicated field values not 'sticking' in clustered POJO

            I'd suggest that you troubleshoot this one step at a time. First is to make sure the real standalone mode (e.g., outside of any container) works first with your POJO. Then you are sure that POJO instrumentation is done correctly.

            The second step will be to turn on all log tracing to debug (yes, lots of output). Thing you want to look for field replication is like replicate(..., put(...)) that is replicating to the other node.

            • 3. Re: Replicated field values not 'sticking' in clustered POJO
              mbenner

              Thank you very much for your responses. Here's what I've found based on your suggestions (sorry for the verboseness of the log statements I've included).

              I start ServerA first and add a MerchantDailyAccount object to it with a 'dailySpend' value of 128.00 (I know it gets added successfully because I have debug logging to print out its values immediately after it gets added to cache). I then start ServerB and add a MerchantDailyAccount object also with a 'dailySpend' value of 128.00.

              This is what I see in the logs on ServerA (192.168.1.174) - the server being replicated to:

              26 Jun 2006 20:14:37,165 DEBUG[] org.jboss.cache.interceptors.PessimisticLockInterceptor - PessimisticLockInterceptor invoked for method prepare(GlobalTransaction:<192.168.1.184:33302>:1, [_put(GlobalTransaction:<192.168.1.184:33302>:1, /test/dailySpend, dummy, dummy, true), _put(GlobalTransaction:<192.168.1.184:33302>:1, /test/dailySpend, AOPInstance, org.jboss.cache.aop.AOPInstance@bbfa5c, true), _put(GlobalTransaction:<192.168.1.184:33302>:1, /test/dailySpend, __jboss:internal:class__, class com.gifts.business.MerchantDailyAccount, true), _put(GlobalTransaction:<192.168.1.184:33302>:1, /test/dailySpend, {dailySpend=128.0}, true), _remove(GlobalTransaction:<192.168.1.184:33302>:1, /test/dailySpend, dummy, true)], 192.168.1.184:33302, false)
              26 Jun 2006 20:14:37,165 DEBUG[] org.jboss.cache.interceptors.PessimisticLockInterceptor - bypassed locking as method prepare() doesn't require locking
              26 Jun 2006 20:14:37,166 DEBUG[] org.jboss.cache.interceptors.CallInterceptor - Suppressing invocation of method prepare(GlobalTransaction:<192.168.1.184:33302>:1, [_put(GlobalTransaction:<192.168.1.184:33302>:1, /test/dailySpend, dummy, dummy, true), _put(GlobalTransaction:<192.168.1.184:33302>:1, /test/dailySpend, AOPInstance, org.jboss.cache.aop.AOPInstance@bbfa5c, true), _put(GlobalTransaction:<192.168.1.184:33302>:1, /test/dailySpend, __jboss:internal:class__, class com.gifts.business.MerchantDailyAccount, true), _put(GlobalTransaction:<192.168.1.184:33302>:1, /test/dailySpend, {dailySpend=128.0}, true), _remove(GlobalTransaction:<192.168.1.184:33302>:1, /test/dailySpend, dummy, true)], 192.168.1.184:33302, false) on cache.


              Seems like the object is being passed correctly across the network, right? However, when I look at the objects on ServerA and ServerB, I see:

              ServerA: MerchantDailyAccount.dailySpend = 1.00 (I should see 128.00, but instead I see this which is the default value)
              ServerB: MerchantDailyAccount.dailySpend = 128.00 (This is what I expect to see)

              I then run a test to modify the 'dailySpend' value of the object on ServerB. When I update its value to 128.55 on ServerB, this is what I see
              in the logs on both servers:

              ServerB (192.168.1.184) - the server being updated directly

              26 Jun 2006 20:18:08,115 DEBUG[] org.jboss.cache.interceptors.ReplicationInterceptor - invoking method _put(null, /test/dailySpend, dailySpend, 128.55, true), members=[192.168.1.174:33961, 192.168.1.184:33302], mode=REPL_SYNC, exclude_self=true, timeout=20000
              26 Jun 2006 20:18:08,115 DEBUG[] org.jboss.cache.TreeCache - callRemoteMethods(): valid members are [192.168.1.174:33961] methods: _put(null, /test/dailySpend, dailySpend, 128.55, true)
              26 Jun 2006 20:18:08,131 DEBUG[] org.jboss.cache.TreeCache - (192.168.1.184:33302): responses for method _replicate:
              [sender=192.168.1.174:33961, retval=128.0, received=true, suspected=false]
              26 Jun 2006 20:18:08,131 DEBUG[] org.jboss.cache.interceptors.ReplicationInterceptor - responses=[128.0]


              ServerA (192.168.1.174) - the server being replicated to

              26 Jun 2006 20:18:08,122 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - (192.168.1.174:33961) call on method [_put(null, /test/dailySpend, dailySpend, 128.55, true)]
              26 Jun 2006 20:18:08,123 DEBUG[] org.jboss.cache.interceptors.ReplicationInterceptor - IsLocalCommitOrRollback=false
              26 Jun 2006 20:18:08,123 DEBUG[] org.jboss.cache.interceptors.PessimisticLockInterceptor - PessimisticLockInterceptor invoked for method _put(null, /test/dailySpend, dailySpend, 128.55, true)
              26 Jun 2006 20:18:08,123 DEBUG[] org.jboss.cache.interceptors.PessimisticLockInterceptor - Attempting to lock node /test/dailySpend for owner Thread[Thread-14,5,main]
              26 Jun 2006 20:18:08,123 DEBUG[] org.jboss.cache.Node - acquiring RL: fqn=/test, caller=Thread[Thread-14,5,main], lock=
              26 Jun 2006 20:18:08,123 DEBUG[] org.jboss.cache.Node - acquired RL: fqn=/test, caller=Thread[Thread-14,5,main], lock=read owners=[Thread[Thread-14,5,main]]
              26 Jun 2006 20:18:08,123 DEBUG[] org.jboss.cache.Node - acquiring WL: fqn=/test/dailySpend, caller=Thread[Thread-14,5,main], lock=
              26 Jun 2006 20:18:08,123 DEBUG[] org.jboss.cache.Node - acquired WL: fqn=/test/dailySpend, caller=Thread[Thread-14,5,main], lock=write owner=Thread[Thread-14,5,main]
              26 Jun 2006 20:18:08,123 DEBUG[] org.jboss.cache.interceptors.CallInterceptor - Invoking method _put(null, /test/dailySpend, dailySpend, 128.55, true) on cache.
              26 Jun 2006 20:18:08,123 DEBUG[] org.jboss.cache.TreeCache - _put(null, "/test/dailySpend", dailySpend, 128.55)
              26 Jun 2006 20:18:08,123 DEBUG[] org.jboss.cache.eviction.RegionManager - nodeModified(): redirecting to node visited. fqn- /test/dailySpend
              26 Jun 2006 20:18:08,123 DEBUG[] org.jboss.cache.eviction.RegionManager - nodeVisited(): fqn- /test/dailySpend


              So, from these logs, it looks like ServerB is sending a value of dailySpend = 128.55 to ServerA, and ServerA is sending a response back indicating it updated the value (meaning that my object is instrumented correctly, right?). I'm not sure why ServerA is sending back a return value of 128.0 except that this is the value that should have existed prior to the replication attempt? This is odd since when I got the object out of ServerA cache prior to running this update test, it told me the value was 1.0.

              So now, when I now look at the objects on the two servers, I see:

              ServerA: MerchantDailyAccount.dailySpend = 1.00 (still stuck at 1.00)
              ServerB: MerchantDailyAccount.dailySpend = 128.55 (correct value)

              I can't seem to correlate what I'm seeing in the logs with what I'm not seeing in the objects. It seems like the values are being
              replicated between the two servers but not actually persisted in the object being replicated to.

              Is there anything else I could be looking for in the logs? I don't see any errors or anything that says an action failed. Any reason why these
              values are being replicated but aren't actually persisting in the objects themselves?

              For your reference, the class I'm caching is:

              @org.jboss.cache.aop.annotation.PojoCacheable
              public class MerchantDailyAccount implements Serializable {
              
               protected float dailySpend = 1.0F;
              
              
               /**
               * Default constructor
               */
               public MerchantDailyAccount() { }
              
              
               public float getDailySpend() {
               return dailySpend;
               }
              
               public void setDailySpend(float dailySpend) {
               this.dailySpend = dailySpend;
               }
              }


              Thanks,
              Michael

              • 4. Re: Replicated field values not 'sticking' in clustered POJO

                Hmmn... looks like replication is happening and you have field replication as well.

                How about you do it the regular way: starting A and B, then do a putObject on A then do a getObject from B?

                Check the log file as well.

                • 5. Re: Replicated field values not 'sticking' in clustered POJO
                  mbenner

                  So, I tried what you suggested, and here's what I see in the logs. In particular, I'm seeing a message on Server B (the server replicated to) that says:

                  Suppressing invocation of method commit(GlobalTransaction:<192.168.1.174:33972>:1) on cache.

                  Could this be the problem, and if so, what could possibly cause this? I've included the entire set of log statements below.


                  27 Jun 2006 13:48:55,107 DEBUG[] org.jboss.cache.interceptors.PessimisticLockInterceptor - PessimisticLockInterceptor invoked for method prepare(GlobalTransaction:<192.168.1.174:33972>:1, [_put(GlobalTransaction:<192.168.1.174:33972>:1, /test/dailySpend, dummy, dummy, true), _put(GlobalTransaction:<192.168.1.174:33972>:1, /test/dailySpend, AOPInstance, org.jboss.cache.aop.AOPInstance@110b640, true), _put(GlobalTransaction:<192.168.1.174:33972>:1, /test/dailySpend, __jboss:internal:class__, class com.gifts.business.MerchantDailyAccount, true), _put(GlobalTransaction:<192.168.1.174:33972>:1, /test/dailySpend, {test_double=0.0, test_boolean=true, testDouble=0.0, testString=null, test_int=0, testBoolean=false, testInteger=0}, true), _remove(GlobalTransaction:<192.168.1.174:33972>:1, /test/dailySpend, dummy, true)], 192.168.1.174:33972, false)
                  27 Jun 2006 13:48:55,107 DEBUG[] org.jboss.cache.interceptors.PessimisticLockInterceptor - bypassed locking as method prepare() doesn't require locking
                  27 Jun 2006 13:48:55,108 DEBUG[] org.jboss.cache.interceptors.CallInterceptor - Suppressing invocation of method prepare(GlobalTransaction:<192.168.1.174:33972>:1, [_put(GlobalTransaction:<192.168.1.174:33972>:1, /test/dailySpend, dummy, dummy, true), _put(GlobalTransaction:<192.168.1.174:33972>:1, /test/dailySpend, AOPInstance, org.jboss.cache.aop.AOPInstance@110b640, true), _put(GlobalTransaction:<192.168.1.174:33972>:1, /test/dailySpend, __jboss:internal:class__, class com.gifts.business.MerchantDailyAccount, true), _put(GlobalTransaction:<192.168.1.174:33972>:1, /test/dailySpend, {test_double=0.0, test_boolean=true, testDouble=0.0, testString=null, test_int=0, testBoolean=false, testInteger=0}, true), _remove(GlobalTransaction:<192.168.1.174:33972>:1, /test/dailySpend, dummy, true)], 192.168.1.174:33972, false) on cache.
                  27 Jun 2006 13:48:55,108 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - Are we running a 1-phase commit? false
                  27 Jun 2006 13:48:55,108 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - Finished remote prepare GlobalTransaction:<192.168.1.174:33972>:1
                  27 Jun 2006 13:48:55,134 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - (192.168.1.184:33307) call on method [commit(GlobalTransaction:<192.168.1.174:33972>:1)]
                  27 Jun 2006 13:48:55,134 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - Got gtx from method call GlobalTransaction:<192.168.1.174:33972>:1
                  27 Jun 2006 13:48:55,135 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - Found local TX=org.jboss.cache.transaction.DummyTransaction@b914b3, global TX=GlobalTransaction:<192.168.1.174:33972>:1
                  27 Jun 2006 13:48:55,135 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - executing commit(GlobalTransaction:<192.168.1.174:33972>:1)() with local TX org.jboss.cache.transaction.DummyTransaction@b914b3 under global tx GlobalTransaction:<192.168.1.174:33972>:1
                  27 Jun 2006 13:48:55,135 DEBUG[] org.jboss.cache.transaction.DummyTransaction - processing beforeCompletion for tx=org.jboss.cache.transaction.DummyTransaction@b914b3, handlers=[TxInterceptor.RemoteSynchronizationHandler(gtx=GlobalTransaction:<192.168.1.174:33972>:1, tx=org.jboss.cache.transaction.DummyTransaction@b914b3)]
                  27 Jun 2006 13:48:55,135 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - Running beforeCompletion on gtx GlobalTransaction:<192.168.1.174:33972>:1
                  27 Jun 2006 13:48:55,135 DEBUG[] org.jboss.cache.transaction.DummyTransaction - processing afterCompletion for tx=org.jboss.cache.transaction.DummyTransaction@b914b3, handlers=[TxInterceptor.RemoteSynchronizationHandler(gtx=GlobalTransaction:<192.168.1.174:33972>:1, tx=org.jboss.cache.transaction.DummyTransaction@b914b3)]
                  27 Jun 2006 13:48:55,135 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - calling aftercompletion for GlobalTransaction:<192.168.1.174:33972>:1
                  27 Jun 2006 13:48:55,136 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - Running commit phase. One phase? false
                  27 Jun 2006 13:48:55,136 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - running commit for GlobalTransaction:<192.168.1.174:33972>:1
                  27 Jun 2006 13:48:55,136 DEBUG[] org.jboss.cache.interceptors.ReplicationInterceptor - IsLocalCommitOrRollback=false
                  27 Jun 2006 13:48:55,136 DEBUG[] org.jboss.cache.interceptors.PessimisticLockInterceptor - PessimisticLockInterceptor invoked for method commit(GlobalTransaction:<192.168.1.174:33972>:1)
                  27 Jun 2006 13:48:55,136 DEBUG[] org.jboss.cache.interceptors.PessimisticLockInterceptor - committing cache with gtx GlobalTransaction:<192.168.1.174:33972>:1
                  27 Jun 2006 13:48:55,136 DEBUG[] org.jboss.cache.interceptors.PessimisticLockInterceptor - releasing lock for /test/dailySpend (write owner=GlobalTransaction:<192.168.1.174:33972>:1)
                  27 Jun 2006 13:48:55,136 DEBUG[] org.jboss.cache.interceptors.PessimisticLockInterceptor - releasing lock for /test (read owners=[GlobalTransaction:<192.168.1.174:33972>:1])
                  27 Jun 2006 13:48:55,137 DEBUG[] org.jboss.cache.interceptors.PessimisticLockInterceptor - removing local transaction org.jboss.cache.transaction.DummyTransaction@b914b3 and global transaction GlobalTransaction:<192.168.1.174:33972>:1
                  27 Jun 2006 13:48:55,137 DEBUG[] org.jboss.cache.interceptors.PessimisticLockInterceptor - bypassed locking as method commit() doesn't require locking
                  27 Jun 2006 13:48:55,137 DEBUG[] org.jboss.cache.interceptors.CallInterceptor - Suppressing invocation of method commit(GlobalTransaction:<192.168.1.174:33972>:1) on cache.
                  27 Jun 2006 13:48:55,137 DEBUG[] org.jboss.cache.interceptors.UnlockInterceptor - Attempting to release locks on current thread. Lock table is {}
                  27 Jun 2006 13:48:55,137 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - Finished local commit/rollback method for GlobalTransaction:<192.168.1.174:33972>:1
                  27 Jun 2006 13:48:55,137 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - Finished commit phase
                  27 Jun 2006 13:48:55,137 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - Resuming suspended transaction null
                  27 Jun 2006 13:48:55,137 DEBUG[] org.jboss.cache.interceptors.TxInterceptor - Finished remote commit/rollback method for GlobalTransaction:<192.168.1.174:33972>:1

                  Thank you so much for your help!
                  Michael

                  • 6. Re: Replicated field values not 'sticking' in clustered POJO
                    brian.stansberry

                    I'm late to this thread, so please forgive if I miss something stated earlier in all of the above :-)

                    First, the "Suppressing invocation" message is completely normal. The cache itself (as opposed to the interceptor stack) doesn't actually do anything on a commit() call, so the CallInterceptor just logs the message you see and discards the call.

                    Second, when you are comparing your objects, where are you getting them from? Do you hold a reference to an existing object, or do you get the object from the cache? If you put a *different* object in the cache under an existing Fqn, this doesn't *update* the existing object, it *replaces* it. Therefore, if other code holds a reference to the old object, it shouldn't expect to see the new object's values.

                    Example unit test code to illustrate my point:

                    MerchantDailyAccount pojoA = new MerchantDailyAccount(1);
                    cacheA.putObject("/test", pojoA);
                    
                    MerchantDailyAccount pojoB = new MerchantDailyAccount(128);
                    cacheB.putObject("/test", pojoB);
                    
                    // pojoB REPLACED pojoA in the cache, so pojoA should
                    // not have pojoB's dailySpend value
                    
                    assertEquals(1, pojoA.dailySpend);
                    
                    // Get the object from cacheA
                    MerchantDailyAccount replacedA = cacheA.getObject("/test");
                    
                    assertEquals(128, replacedA.dailySpend);
                    
                    // Do an update of the existing object, not a replacement
                    pojoB.setDailySpend(128.5);
                    
                    assertEquals(128.5, replacedA.dailySpend);
                    


                    • 7. Re: Replicated field values not 'sticking' in clustered POJO
                      mbenner

                      Hi Brian,

                      Thanks for your response (and sorry for the verboseness of the previous posts). =)

                      I'm actually getting the object from the cache each time using the getObject() call.

                      // Put object in CacheA and then replace it in CacheB
                      MerchantDailyAccount pojoA = new MerchantDailyAccount(1); // dailySpend=1
                      cacheA.putObject("/test", pojoA);
                      
                      MerchantDailyAccount pojoB = new MerchantDailyAccount(900); // dailySpend=900
                      cacheB.putObject("/test", pojoB);
                      
                      
                      // Re-retrieve the object from CacheA
                      MerchantDailyAccount cachedPojoA = cacheA.getObject("/test");
                      
                      // This test always fails for me. The value of dailySpend in pojoA
                      // actually turns out to be 0, the default value in the object
                      assertEquals(1, pojoA.dailySpend);
                      
                      
                      // Then update the object on CacheA
                      pojoA.setDailySpend(128.0);
                      
                      
                      // Re-retrieve the POJOs from CacheA and CacheB
                      MerchantDailyAccount updatedPojoA = cacheA.getObject("/test");
                      MerchantDailyAccount updatedPojoB = cacheB.getObject("/test");
                      
                      
                      assertEquals(128.0, pojoA.dailySpend); // this one will pass
                      assertEquals(128.0, pojoB.dailySpend); // this one will always fail. dailySpend will equal 900


                      I see the correct values being passed from CacheA to CacheB in the logs (and vice versa). The values just aren't being populated in the objects.

                      The really odd thing that I can consistently produce is that instead of updating the dailySpend by calling setDailySpend() (as in the code above), if I just increment the value by 0.50, for example, via an internal method in MerchantDailyAccount, it'll actually change the value to 900.5, which is what I would expect. So, it seems the replicated values are actually getting across, they just aren't getting assigned to the fields in the object.

                      I'm going to try JBoss Cache 1.4 and see if that fixes it. Otherwise, I'm stumped.

                      Thanks!
                      Michael


                      • 8. Re: Replicated field values not 'sticking' in clustered POJO
                        brian.stansberry

                        No worries about long posts if the problem is complex :-) I was just too lazy to read all the details.

                        I'm not sure if this was just a typo in the forum post, but in what you posted the variable you assign when you call getObject() is not the one you are using in your assertions. "cachedPojoA" is never used again, and the call to "pojoA.setDailySpend(128.0)" is on an object no longer in the cache.

                        I think this was a typo, but want to be sure.

                        Even if it wasn't a typo that wouldn't explain why "assertEquals(1, pojoA.dailySpend);" fails. Nor the anomaly of why adding .5 is OK but a direct set is not.

                        • 9. Re: Replicated field values not 'sticking' in clustered POJO

                          michael,

                          Sorry that it wasn' clear before but I probably know where is the problem. You are doing:

                          cacheA.putObject("/test", pojoA); // pojo attached on cache A
                          
                          cacheB.putObject("/test", pojoB); // pojo attached on cache B as well
                          
                          pojoA.setBlah(xx); // will have problem here
                          


                          Any reason doing simultaneous pojo putObject from both nodes? This is a problem that currently PojoCache don't handle. I have opened a Jira (JBCACHE-674) to track this.

                          Alternative is to use different fqns for different POJOs. Remember that POJO lifetime is not supposed to be short-lived, otherwise, PojoCache won't be effective.

                          Just to confirm, the regular usage:

                          cacheA.putObject("/test", pojoA); // pojo attached on cache A
                          
                          pojoA1 = cacheB.getObject("/test");
                          
                          pojoA.setBlah(xx); // will have problem here
                          print(pojoA1); will get updated.
                          

                          will work, correct?


                          • 10. Re: Replicated field values not 'sticking' in clustered POJO
                            mbenner

                            Ben and Brian,

                            First off, I'm sorry for the typo in my previous response. When running my assertEquals() tests, I was actually running it on the re-retrieved POJO objects (cachedPojoA, updatedPojoA, and updatedPojoB). Thank you for catching that typo. :)

                            Good to know that doing a putObject() from both nodes isn't supported. That explains part of the problem I'm seeing, but unfortunately not all. If I follow the regular usage pattern, as you described:

                            cacheA.putObject("/test", pojoA); // pojo attached on cache A
                            
                            pojoA1 = cacheB.getObject("/test");
                            
                            pojoA.setBlah(xx); // will have problem here
                            print(pojoA1); //will get updated.


                            pojoA1 won't exist on cacheB (I'll get a null value in this test). In the logs, I'll see the object and values coming over to CacheB, but it won't actually get set into CacheB.

                            As for the reason I'm doing a putObject() from both nodes ... each night, the servers corresponding with CacheA and CacheB restart (various reasons behind this) in about a 30 second staggered fashion. For this particular 'dailySpend' data, we need to restore it from the database on server startup (lazy loading isn't really an option here). One of the two servers needs to load it from the database at startup, and because I'm never entire sure which one will complete its startup first, I'm just having them both load the data from the DB. I'm also afraid to code one server to be the primary DB loader (e.g., CacheA) and the other to just retrieve the data across the network (e.g., CacheB) from the other node because, if CacheA went down permanently, I'd have no way of loading the data from the DB. I'm not sure if there's a way around this (any suggestions would be appreciated) - I looked into your CacheLoaders, but they didn't seem to fit. Perhaps I'm missing something??

                            Anyway, I still need figure out why replicated values aren't getting persisted on the various nodes. :) I'm going to try JBoss Cache 1.4 and see if that give me any different results. Could this be a configuration issue??

                            Thanks,
                            Michael

                            • 11. Re: Replicated field values not 'sticking' in clustered POJO
                              brian.stansberry

                              So if you do:

                              cacheA.putObject("/test", pojoA); // pojo attached on cache A

                              pojoA1 = cacheB.getObject("/test");

                              pojoA1 is null? You've definitely got something weird going on then. That's just basic stuff. Not sure what to say about that :( Hopefully Ben will have some idea.

                              Well, let's just pretend that issue is solved. :-) Re: the JBCACHE-674 problem Ben mentioned with calling:

                              cacheA.putObject("/test", pojoA);
                              cacheB.putObject("/test", pojoB);

                              A workaround should be to just do this:

                              cacheA.getObject("/test");
                              cacheA.putObject("/test", pojoA);

                              cacheB.getObject("/test");
                              cacheB.putObject("/test", pojoB);

                              If you call getObject() on the 2nd cache before replacing the object, that's enough to put the cache into a state where it detects that an object replacement has happened. (Of course you call getObject() on both caches, because you don't know which one will be the "2nd" cache).

                              • 12. Re: Replicated field values not 'sticking' in clustered POJO
                                mbenner

                                Hi. Yeah, if I do:

                                cacheA.putObject("/test", pojoA);
                                pojoA1 = cacheB.getObject("/test");

                                pojoA1 would be null. With 1.4, however, pojoA1 is showing up, but its empty (the values I assign to it before putting it in cacheA aren't there). Very odd behavior, especially since I see the correct values in the logs.

                                I'll try your suggestion about calling getObject() before putObject(). Crossing fingers...

                                • 13. Re: Replicated field values not 'sticking' in clustered POJO

                                  On the first problem of your requirement, Brian has pretty much cover it. The point is, if you don't care much about the exisiting POJO, then you can do putObject on either node. Just to make sure, to do getObject() before you start doing POJO update, e.g.,

                                  pojoA = cacheA.getObject("/test");
                                  pojoA.setBlah(xx); // should work
                                  


                                  or
                                  pojoB = cacheB.getObject("/test");
                                  pojoB.setBlah(xx); // should work
                                  


                                  As for the first basic problem, I'd recommend that you try out the examples that come with the distribution. You can simply try it out and then modify from there to see if that works for you.

                                  If it works, then probably you need to create a test case and send it to me to look at. We can do it via Jira.