2 Replies Latest reply on Aug 23, 2018 9:10 PM by Purushothaman S

    Use Deny_read_writes and Preferred_always in replicated cache

    Purushothaman S Newbie



      I have few questions on the partition handling and the merge conflict functionality in the latest Infinispan release (9.3.1). We are planning to deploy the latest Infinispan server in cluster mode(size 3) with replicated caches. Our application requires data consistency over availability hence we configured the partition handling policy to 'DENY_READ_WRITES' in all our replicated cache.


      Can you please suggest on the merge policy to configure on our replicated caches? We configured it with MergePolicy.NONE. It seemed to work and the Infinispan servers were able to join back the cluster upon performing a graceful restart of 2 servers in the cluster. However, when we did an ungraceful shutdown of non coordinator and coordinator(in that order), the servers don't join back the cluster and we see the following exception in our application:


      03-Aug-2018 14:33:25,193 ERROR [com.alu.cna.cloudmgmt.cache.beans.impl.CacheManagerWrapperBean] (pool-8-thread-1) <:> Cache health check operation failed.

      com.alu.cna.cloudmgmt.cache.exceptions.CacheOperationFailedException: org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=39796 returned server error (status=0x85): org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'WrappedByteArray{bytes=[B0x033E257673642D32..[40], hashCode=0}' is not available. Not all owners are in this partition


      Caused by: org.infinispan.client.hotrod.exceptions.HotRodClientException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'WrappedByteArray{bytes=[B0x033E257673642D32..[40], hashCode=0}' is not available. Not all owners are in this partition

              at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:333) ~[infinispan-client-hotrod-9.3.1.Final.jar:9.3.1.Final]

              at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:179) ~[infinispan-client-hotrod-9.3.1.Final.jar:9.3.1.Final]

              at org.infinispan.client.hotrod.impl.transport.netty.HeaderDecoder.decode(HeaderDecoder.java:138) ~[infinispan-client-hotrod-9.3.1.Final.jar:9.3.1.Final]

              at org.infinispan.client.hotrod.impl.transport.netty.HintedReplayingDecoder.callDecode(HintedReplayingDecoder.java:98) ~[infinispan-client-hotrod-9.3.1.Final.jar:9.3.1.Final]

              at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.22.Final.jar:4.1.22.Final]

              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]


      Also, the following errors were seen in Infinispan server log



      2018-08-03 14:33:25,191 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-hotrod-abc-connector-ServerWorker-3-4) ISPN000136: Error executing command PutKeyValueCommand, writing keys [WrappedByteArray{bytes=[B0x033E257673642D32..[40], hashCode=0}]: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'WrappedByteArray{bytes=[B0x033E257673642D32..[40], hashCode=0}' is not available. Not all owners are in this partition

              at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:258)

              at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkWrite(PartitionHandlingManagerImpl.java:96)

              at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleSingleWrite(PartitionHandlingInterceptor.java:89)

              at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitPutKeyValueCommand(PartitionHandlingInterceptor.java:59)

              at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:68)


      2018-08-03 14:14:29,092 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service jboss.datagrid-infinispan.abc.default: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan.abc.default: Failed to start service

              at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1728)

              at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1556)

              at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)

              at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)

              at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)

              at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)

              at java.lang.Thread.run(Unknown Source)

      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl

              at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)


      Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on host-2.abc.com

              at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

              at java.lang.reflect.Method.invoke(Unknown Source)

              at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)


      Should we configure the merge policy as PREFERRED_ALWAYS? As I understand, in a replicated cache the keys are owned by all the nodes and one of them being designated as primary. With PREFERRED_ALWAYS the value in the primary owner is picked during a merge conflict. Would configuring DENY_READ_WRITES and PREFERRED_ALWAYS cause any data consistency issues during a split brain / ungraceful shutdown in a replicated cache?




        • 1. Re: Use Deny_read_writes and Preferred_always in replicated cache
          Radim Vansa Master

          So, to explain merge-policy vs. consistency: With DENY_READS_WRITES Infinispan tries to keep cache consistent, but it's not always possible due to Two Generals' Problem (wiki that). So when the split occurs during a write it is possible that the value on different owners diverges, and there's nothing you could do about it. Such inconsistency may happen even during regular operation (without partition) e.g. when some timeouts are exceeded. Basically when your operation ends with an exception, there's a chance that the owners will be inconsistent. So you need to code your application so that it can live in such state (providing wrong data or failing some operations gracefully and not crashing completely).


          When the cluster joins back together, there's the reconciliation process that tries to fix these inconsistencies - make all owners hold the same value. By configuring MergePolicy.NONE you disable this process. It's up to you if you don't mind that cache.get() may return different values, but I guess that you're better off eventually getting into consistent state. ryanemerson When the cluster joins, does the merge happen before you return availability status to available, or after that?


          So about the AvailabilityExceptions: you're expected to see them during partition and for a while until the partition heals. However the 'CacheException: Initial state transfer timed out for cache default on host-2.abc.com' is weird; have you configured the state-transfer timeout to a different value than the default (4 minutes IIRC)? How much data do you have in the cache and what are the entry sizes?

          • 2. Re: Use Deny_read_writes and Preferred_always in replicated cache
            Purushothaman S Newbie

            Thanks rvansa We have configured a state-transfer timeout of 5 minutes in all our replicated-cache instances. As I remember, the cache was either empty (or) had very few entries. Please let me know if you want the issue to be reproduced the issue with the required log levels.