I am accessing infinispan data grid 5.3 in clustered mode (setup on 2 nodes) from my application hosted on EAP 6.1.1.

While carrying out following test I am getting an exception.

1. Start both hot rod servers.

2. Access servers using application hosted on EAP [infinispan.client.hotrod.server_list=HOST1:11222;HOST2:11222]. This is the only property defined in properties file, rest are default.

3. Cache operations working fine so far

4. Shutdown HOST1.

5. Cache operations starts giving following exception, I was expecting the cache operations to work successfully on HOST2

17:00:07,450 WARN  [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (http-localhost/127.0.0.1:8080-1) ISPN004022: Unable to invalidate transport for server: /HOST1:11222

17:00:08,739 ERROR [org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory] (http-localhost/127.0.0.1:8080-1) ISPN004017: Could not fetch transport: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: /HOST1:11222

  at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:88)

  at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:57)

  at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:38)

  at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)

  at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:271)

  at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:149)

  at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:63)

  at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:67)

  at org.infinispan.client.hotrod.impl.RemoteCacheImpl.get(RemoteCacheImpl.java:353)

Notes:

- Cache operations works fine in case where HOST2 is down and HOST1 is up. So it seems whichever host we define first in server_list if it goes down it could not recover from this excpetion until we make that server up. 

- There is only one instance of RemoteCacheManager for this application which is initialized only once.

Appreciate any inputs to resolve this issue.

-Nikhil