We are trying to run Infinispan with number of owners as 2 and number of instances > 2.

When owners and instances are equal there is no issue but when it becomes different we are facing some marshalling related issue as given below.

Error stack is

ERROR [org.infinispan.interceptors.InvocationContextInterceptor] [HotRodServerWorker-3] (InvocationContextInterceptor.java:130) - ISPN000136: Execution error

java.lang.ClassCastException: [B cannot be cast to org.infinispan.container.entries.CacheEntry

        at org.infinispan.interceptors.compat.BaseTypeConverterInterceptor.visitGetKeyValueCommand(BaseTypeConverterInterceptor.java:78)

        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)

        at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)

        at org.infinispan.CacheImpl.getCacheEntry(CacheImpl.java:384)

        at org.infinispan.DecoratedCache.getCacheEntry(DecoratedCache.java:496)

        at org.infinispan.server.core.AbstractProtocolDecoder.get(AbstractProtocolDecoder.scala:259)

        at org.infinispan.server.core.AbstractProtocolDecoder.decodeKey(AbstractProtocolDecoder.scala:99)

        at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:53)

        at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:29)

        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)

        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)

        at org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:377)

        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)

        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)

        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)

        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:724)

WARN [org.infinispan.client.hotrod.impl.protocol.Codec12] [Task 6014] (Codec10.java:141) - ISPN004005: Error received from the server: java.lang.ClassCastException: [B cannot be cast to org.infinispan.container.entries.CacheEntry

Configuration is : ConfigurationBuilder{classLoader=java.lang.ref.WeakReference@337e9c58, clustering=ClusteringConfigurationBuilder{async=AsyncConfigurationBuilder{asyncMarshalling=false, replicationQueue=null, replicationQueueInterval=5000, replicationQueueMaxElements=1000, useReplicationQueue=false}, cacheMode=DIST_SYNC, hash=HashConfigurationBuilder{consistentHashFactory=null, hash=MurmurHash3, numOwners=2, numSegments=100, groups=GroupsConfigurationBuilder{enabled=false, groupers=[]}}, l1=L1ConfigurationBuilder{enabled=false, invalidationThreshold=0, lifespan=600000, cleanupTaskFrequency=600000, onRehash=false}, stateTransfer=StateTransferConfigurationBuilder{chunkSize=1000, fetchInMemoryState=true, awaitInitialTransfer=true, timeout=3600000}, sync=SyncConfigurationBuilder{replTimeout=15000}}, customInterceptors=CustomInterceptorsConfigurationBuilder{interceptors=[]}, dataContainer=DataContainerConfigurationBuilder{dataContainer=null, properties={}, keyEquivalence=org.infinispan.commons.equivalence.AnyEquivalence@3d4a6fd1, valueEquivalence=org.infinispan.commons.equivalence.AnyEquivalence@3d4a6fd1}, deadlockDetection=DeadlockDetectionConfigurationBuilder{enabled=false, spinDuration=100}, eviction=EvictionConfigurationBuilder{maxEntries=-1, strategy=NONE, threadPolicy=DEFAULT}, expiration=ExpirationConfigurationBuilder{lifespan=-1, maxIdle=-1, reaperEnabled=true, wakeUpInterval=60000}, indexing=IndexingConfigurationBuilder{enabled=false, indexLocalOnly=false, properties={}}, invocationBatching=InvocationBatchingConfigurationBuilder{enabled=false}, jmxStatistics=JMXStatisticsConfigurationBuilder{enabled=true}, persistence=PersistenceConfigurationBuilder{stores=[com.subex.spark.common.distributedcaching.server.cachestore.IspnCacheStoreConfigurationBuilder@61f321c6], passivation=false}, locking=LockingConfigurationBuilder{concurrencyLevel=32, isolationLevel=READ_COMMITTED, lockAcquisitionTimeout=10000, useLockStriping=false, writeSkewCheck=false}, modules=[], storeAsBinary=StoreAsBinaryConfigurationBuilder{enabled=false, storeKeysAsBinary=true, storeValuesAsBinary=true}, transaction=TransactionConfigurationBuilder{autoCommit=true, cacheStopTimeout=30000, eagerLockingSingleNode=false, lockingMode=OPTIMISTIC, syncCommitPhase=true, syncRollbackPhase=false, transactionManagerLookup=org.infinispan.transaction.lookup.GenericTransactionManagerLookup@212e8067, transactionSynchronizationRegistryLookup=null, transactionMode=TRANSACTIONAL, useEagerLocking=false, useSynchronization=true, recovery=RecoveryConfigurationBuilder{enabled=true, recoveryInfoCacheName='__recoveryInfoCacheName__'}, use1PcForAutoCommitTransactions=false, completedTxTimeout=15000, reaperWakeUpInterval=1000}, versioning=VersioningConfigurationBuilder{enabled=true, scheme=NONE}, unsafe=UnsafeConfigurationBuilder{unreliableReturnValues=false}, sites=org.infinispan.configuration.cache.SitesConfigurationBuilder@73effb35, compatibility=org.infinispan.configuration.cache.CompatibilityModeConfigurationBuilder@1055900a}