Use Deny_read_writes and Preferred_always in replicated cache

[purushos]

Hi,

I have few questions on the partition handling and the merge conflict functionality in the latest Infinispan release (9.3.1). We are planning to deploy the latest Infinispan server in cluster mode(size 3) with replicated caches. Our application requires data consistency over availability hence we configured the partition handling policy to 'DENY_READ_WRITES' in all our replicated cache.

Can you please suggest on the merge policy to configure on our replicated caches? We configured it with MergePolicy.NONE. It seemed to work and the Infinispan servers were able to join back the cluster upon performing a graceful restart of 2 servers in the cluster. However, when we did an ungraceful shutdown of non coordinator and coordinator(in that order), the servers don't join back the cluster and we see the following exception in our application:

03-Aug-2018 14:33:25,193 ERROR [com.alu.cna.cloudmgmt.cache.beans.impl.CacheManagerWrapperBean] (pool-8-thread-1) <:> Cache health check operation failed.

com.alu.cna.cloudmgmt.cache.exceptions.CacheOperationFailedException: org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=39796 returned server error (status=0x85): org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'WrappedByteArray{bytes=[B0x033E257673642D32..[40], hashCode=0}' is not available. Not all owners are in this partition

.....

Caused by: org.infinispan.client.hotrod.exceptions.HotRodClientException: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'WrappedByteArray{bytes=[B0x033E257673642D32..[40], hashCode=0}' is not available. Not all owners are in this partition

        at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:333) ~[infinispan-client-hotrod-9.3.1.Final.jar:9.3.1.Final]

        at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:179) ~[infinispan-client-hotrod-9.3.1.Final.jar:9.3.1.Final]

        at org.infinispan.client.hotrod.impl.transport.netty.HeaderDecoder.decode(HeaderDecoder.java:138) ~[infinispan-client-hotrod-9.3.1.Final.jar:9.3.1.Final]

        at org.infinispan.client.hotrod.impl.transport.netty.HintedReplayingDecoder.callDecode(HintedReplayingDecoder.java:98) ~[infinispan-client-hotrod-9.3.1.Final.jar:9.3.1.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.22.Final.jar:4.1.22.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]

Also, the following errors were seen in Infinispan server log

2018-08-03 14:33:25,191 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (HotRod-hotrod-abc-connector-ServerWorker-3-4) ISPN000136: Error executing command PutKeyValueCommand, writing keys [WrappedByteArray{bytes=[B0x033E257673642D32..[40], hashCode=0}]: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'WrappedByteArray{bytes=[B0x033E257673642D32..[40], hashCode=0}' is not available. Not all owners are in this partition

        at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:258)

        at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkWrite(PartitionHandlingManagerImpl.java:96)

        at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleSingleWrite(PartitionHandlingInterceptor.java:89)

        at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitPutKeyValueCommand(PartitionHandlingInterceptor.java:59)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:68)

2018-08-03 14:14:29,092 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service jboss.datagrid-infinispan.abc.default: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan.abc.default: Failed to start service

        at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1728)

        at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1556)

        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)

        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)

        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)

        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)

        at java.lang.Thread.run(Unknown Source)

Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl

        at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)

.....

Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on host-2.abc.com

        at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

        at java.lang.reflect.Method.invoke(Unknown Source)

        at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)

Should we configure the merge policy as PREFERRED_ALWAYS? As I understand, in a replicated cache the keys are owned by all the nodes and one of them being designated as primary. With PREFERRED_ALWAYS the value in the primary owner is picked during a merge conflict. Would configuring DENY_READ_WRITES and PREFERRED_ALWAYS cause any data consistency issues during a split brain / ungraceful shutdown in a replicated cache?

Thanks,

Purush

[rvansa]

So, to explain merge-policy vs. consistency: With DENY_READS_WRITES Infinispan tries to keep cache consistent, but it's not always possible due to Two Generals' Problem (wiki that). So when the split occurs during a write it is possible that the value on different owners diverges, and there's nothing you could do about it. Such inconsistency may happen even during regular operation (without partition) e.g. when some timeouts are exceeded. Basically when your operation ends with an exception, there's a chance that the owners will be inconsistent. So you need to code your application so that it can live in such state (providing wrong data or failing some operations gracefully and not crashing completely).

When the cluster joins back together, there's the reconciliation process that tries to fix these inconsistencies - make all owners hold the same value. By configuring MergePolicy.NONE you disable this process. It's up to you if you don't mind that cache.get() may return different values, but I guess that you're better off eventually getting into consistent state. ryanemerson When the cluster joins, does the merge happen before you return availability status to available, or after that?

So about the AvailabilityExceptions: you're expected to see them during partition and for a while until the partition heals. However the 'CacheException: Initial state transfer timed out for cache default on host-2.abc.com' is weird; have you configured the state-transfer timeout to a different value than the default (4 minutes IIRC)? How much data do you have in the cache and what are the entry sizes?

[purushos]

Thanks rvansa We have configured a state-transfer timeout of 5 minutes in all our replicated-cache instances. As I remember, the cache was either empty (or) had very few entries. Please let me know if you want the issue to be reproduced the issue with the required log levels.