6 Replies Latest reply on Aug 4, 2008 4:32 PM by manik

    Compensation for network latency of sync'd DB

    kblanken

      We're about to deploy our application using JBC. We have several systems that work on two databases: The master DB that's always up to date and the slave DB that is synchronized <5 seconds after the master DB. We use DB queue replication.

      All systems use the same cache. We do not always use replication and have to use invalidation quite a lot. In this case, a cache will reload the data from the DB of the system during the next request.

      Now with JBC we might run into the following problem:
      - A system changes the master DB and invalidates the corresponding cache entries.
      - JBC multicasts the invalidation event to the cache cluster members.
      - On the system using the slave DB, the cache entry is invalidated immediately. The DB is not synchronized yet.
      - On the system using the slave DB, somebody requests the value from the cache that has just been invalidated. As the cache doesn't have the data, it loads it from the DB - which means the old version before the update is loaded!
      - The cache contains the old data.
      - The DB synchronization happens, but the cache doesn't know and keeps the outdated data.

      Is there a common solution to this kind of problem?

      My idea was to create a JGroups protocol similar to DELAY (let's say FIXED_DELAY) that sits in the slave machines' JGroups stack and delays all invalidation events for some time to compensate for the DB synchronization lag. How do I know which events or messages I should delay, and where in the stack should I put the protocol?

      Could an ExtendedTreeCacheListener with a delay in nodeRemove help?

      Regards,
      Kai

        • 1. Re: Compensation for network latency of sync'd DB
          genman

          What would be wrong with setting an eviction policy on the cluster so that entries are evicted older than 5 seconds? That would probably be the easiest to configure.

          Though, your delay approach would probably work. Just be sure to have the removal action queued and take place in a separate thread. Some sort of batched remove might work faster.

          • 2. Re: Compensation for network latency of sync'd DB
            kblanken

            Thank you for your answer, genman.
            Evicting every 5 seconds would kill our performance, as we would rarely have a cache hit.

            I've managed to get my FIXED_DELAY protocol to work. It sits on the very top of the stack and delays all Messages with getBuffer()!=null. Unit tests look good so far, I guess we'll try it this way.

            • 3. Re: Compensation for network latency of sync'd DB
              genman

              Your protocol fix sounds like a pretty good solution, although I would be worried if certain messages (not related to invalidation) were held up.

              Could you submit your FIXED_DELAY protocol as patch? https://jira.jboss.org/jira/browse/JGRP It's not required but may be good to have your code examined by the experts.

              • 4. Re: Compensation for network latency of sync'd DB
                manik

                How does your implementation know which messages to delay? Or do you simply delay all messages? You'd want to be careful here since the only way I can think of to tell which messages to delay on a JGroups level is to deserialize everything and then look at the method ids. And delaying everything is not good since join/leave messages could be delayed as well.

                Your best bet may be to write an Interceptor and add this to the start of the chain. You could create a Queue to hold all InvalidateCommands (2.2.0 and above) or MethodCalls containing the invalidate methodId (pre-2.2.0), add them to a DelayQueue and don't pass the call up the interceptor stack any further.

                You could then have another Thread that polls the delay queue grab stuff off the queue as they mature, and then pass them up the interceptor stack.

                • 5. Re: Compensation for network latency of sync'd DB
                  kblanken

                  Hi Manik,

                  I also thought that GMS messages shouldn't be delayed. My assumption from browsing through the source and debugging was that JGroups messages don't use the Message's buffer, but JBC messages do. So my condition for delaying is

                  if (msg.getBuffer()!=null && msg.getBuffer().length>0)

                  Would that be safe? If it is not, I'll have to use the Interceptor as you proposed.

                  • 6. Re: Compensation for network latency of sync'd DB
                    manik

                    It *may* be safe based on your configuration, but either way it would certainly be brittle. E.g., if you use buddy replication, JBC uses the same RPC mechanism to assign buddies, etc. And there is no good reason to delay these, and your approach won't be able to tell the difference.

                    The interceptor chain is the proper extension point for such things since you have a definitive idea of the command/methodcall being invoked (including payload, etc).