I am using Infinispan 7.2.1.Final.

I have a 2 node cluster using the attached jgroups file.

Each node is configured as single owner on my cache entries. I use transactions to ensure write locks per cache key.

I build the cache like this:

GlobalConfiguration globalConfig = new GlobalConfigurationBuilder().transport().defaultTransport().clusterName("apc-cluster").addProperty("configurationFile", "jgroups-tcp.xml")
    .build();
  DefaultCacheManager manager = new DefaultCacheManager(globalConfig)


  AdvancedCache sessionsByHouseholdCache = getSessionCache()
  
  private Cache<String, SessionCacheEntry>  getSessionCache() = {
  Configuration config = new ConfigurationBuilder()
      .read(manager.getDefaultCacheConfiguration)
      .transaction().transactionManagerLookup(new JBossStandaloneJTAManagerLookup()).transactionMode(TransactionMode.TRANSACTIONAL).lockingMode(LockingMode.PESSIMISTIC)
      .clustering().cacheMode(CacheMode.DIST_SYNC).hash().numOwners(1)      
      .build();
    manager.defineConfiguration(SESSION_CACHE, config)
    return manager.getCache(SESSION_CACHE).asInstanceOf<AdvancedCache<String, SessionCacheEntry>>
  }

Whenever I use the cache I run this code:

sessionsByHouseholdCache.getAdvancedCache.getTransactionManager.begin();


sessionCacheEntry = sessionsByHouseholdCache.getAdvancedCache.withFlags(FORCE_WRITE_LOCK).get(hhId)
            try {
              //some logic
            }
            finally {
              sessionsByHouseholdCache.getAdvancedCache.getTransactionManager.commit();
            }    

There is on use case where I want to check if the cache contains my key so I use:

if (!sessionsByHouseholdCache.containsKey(hhId)) {
       //throw 404
}

without beginning transaction.

I have an active/active REST server. If I create the cache on node 1, and then I stop node 1 and try to read the cache in node 2 - instead of containsKey returning false it fails on this:

[Exception: org.infinispan.remoting.RpcException: No more valid responses.  Received invalid responses from all of [Sender{address=IP2-12509, responded=true}]
  at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$FutureCollator.getResponseList(CommandAwareRpcDispatcher.java:650)
  at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:487)
  at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:151)
  at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:569)
  at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287)
  at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.invokeClusterGetCommandRemotely(BaseDistributionInterceptor.java:177)
  at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.retrieveFromRemoteSource(BaseDistributionInterceptor.java:165)
  at org.infinispan.interceptors.distribution.TxDistributionInterceptor.remoteGet(TxDistributionInterceptor.java:362)
  at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitGetCommand(TxDistributionInterceptor.java:157)
  at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitGetKeyValueCommand(TxDistributionInterceptor.java:126)
  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
  at org.infinispan.interceptors.EntryWrappingInterceptor.visitDataReadCommand(EntryWrappingInterceptor.java:130)
  at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:120)
  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
  at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:70)
  at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:70)
  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
  at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
  at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86)
  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
  at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:346)
  at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:318)
  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
  at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
  at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86)
  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
  at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:369)
  at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:354)
  at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86)
  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
  at org.infinispan.interceptors.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:103)
  at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:91)
  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
  at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
  at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
  at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86)
  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
  at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
  at org.infinispan.cache.impl.CacheImpl.containsKey(CacheImpl.java:398)
  at org.infinispan.cache.impl.CacheImpl.containsKey(CacheImpl.java:391)

Am I doing something wrong here?