4 Replies Latest reply on Mar 26, 2013 9:16 AM by sebastiantusk

    putForExternalRead doesn't write to owner

    sebastiantusk

      I have a transactional cache and use putForExternalRead. For some reason I do not see that the PutKeyValueCommand is ever sent to the owners. Only the local cache sees this command. With a non transactional cache the command is sent directly to the owners in NonTxDistributionInterceptor.handleLocalWrite.

       

      My actual problem is that this also means that the owner never registers that it has to send out L1 invalidation messages. If the commands would have been send correctly the owner would register the node where I called putForExternalRead in TxDistributionInterceptor.visitPutKeyValueCommand. What happens now is that I have stale cache entry which never get updated until another local write operation.

       

      I use Infinispan 5.2.1 with a fix for ISPN-2913 and the following cache configuration.

          <namedCache name="entity">

              <jmxStatistics enabled="true" />

             

              <clustering mode="dist">

                  <stateTransfer fetchInMemoryState="false" timeout="20000" />

                  <async />

                  <l1 enabled="true" />

                  <hash numOwners="1"/>

              </clustering>

              <locking isolationLevel="READ_COMMITTED"

                  lockAcquisitionTimeout="15000" useLockStriping="false" />

                 

              <eviction maxEntries="10000" strategy="LRU" />

              <expiration maxIdle="100000" wakeUpInterval="5000"/>

              <storeAsBinary storeKeysAsBinary="true" storeValuesAsBinary="false" enabled="false" />

             

              <transaction transactionMode="TRANSACTIONAL" autoCommit="false" lockingMode="OPTIMISTIC"/>

          </namedCache>

        • 1. Re: putForExternalRead doesn't write to owner
          cyron

          Hello, Sebastian.

          from the javadoc of putForExternalRead, invalidation does not take place.

          you can also reference this.

          https://community.jboss.org/thread/201086

          • 2. Re: putForExternalRead doesn't write to owner
            sebastiantusk

            While putForExternalRead doesn't invalidate anything, the entry stored with putForExternalRead has to be invalidated by other write operations like Cache.put or Cache.remove. That usually works just fine and as expected. To cite and modify the example from the link you posted.

            1. Node A reads X from some data store, then calls cache.putForExternalRead(X,Y), pushing some distinct object X into the cache.
            2. Node B then calls cache.put(X,Z).
            3. Node A's copy of X is invalidated.

            This is what usually happens and would be correct behaviour. What actually happens is the following.

            1. Node A reads X from some data store, then calls cache.putForExternalRead(X,Y), pushing some distinct object X into the cache.
            2. Node B then calls cache.put(X,Z).
            3. Node A's copy of X is never invalidated.

            Here something goes wrong in step 1. Node B should register that Node A has an entry by a call to L1ManagerImpl.addRequestor and use this to invalidate Node A in step 2.

            • 3. Re: putForExternalRead doesn't write to owner
              dan.berindei

              Hi Sebastian

               

              cache.putForExternalRead(k, v) should be replicated on the key k's owners as well, and the originator should be registered as an L1 "requestor" for key k. However I did a quick check in the Infinispan test suite and I didn't find any test for putForExternalRead in distributed mode. Would you mind creating an issue in JIRA for this?

               

              As a workaround, you could set your L1 invalidationThreshold to "0", so that L1 invalidations are always broadcasted to the entire cluster.

              • 4. Re: putForExternalRead doesn't write to owner
                sebastiantusk

                Hi Dan,

                thanks for your answer. I submitted ISPN-2964. Your workaround could indeed help with this specific problem. Unfortunately I have other issues with putForExternalRead, L1 and invalidation. See ISPN-2913 and ISPN-2965. For ISPN-2913 and ISPN-2964 I have fixes but for the third issue I do not have a clue how to approach it. The problem seems to be that writing an entry sends out invalidations but leaves after that a gap where other nodes can retrieve old values. These old values are put into L1 and not invalidated. Leading to a state where one node returns current values and the other one out-dated values. Something like this:

                1. Prepare cache by adding an entry with Cache.put( k, v1 ).
                2. Node B starts with adding a changed value. Cache.put( k, v2 )
                3. Node B TxDistributionInterceptor.visitPrepareCommand - flushL1Caches sends invalidations.
                4. Node A calls Cache.get( k ) retrieves v1 and stores this value in L1.
                5. Node B proceeds with transaction.
                6. A has v1. B has v2.