Replicate cluster cache state transfer issue:: Timeout Exception

[ankitkrsingh]

Hi Team,

I have 2 infinispan 6.0 servers XXXXXXXXX6498 and XXXXXXXXX6499 which are forming cluster between them and cache type is replicated.

I am getting exception

org.infinispan.commons.CacheException: Initial state transfer timed out for cache

during state transfer.

Scenario:

1. Started XXXXXXXXX6498 cache using command ./cluster.sh

2. Started XXXXXXXXX6499 cache using command ./cluster.sh

They formed the cluster which can be seen in the log

14:22:22,148 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000094: Received new cluster view: [XXXXXXXXX6498/clustered|1] (2) [XXXXXXXXX6498/clustered, XXXXXXXXX6499/clustered]

3. Shutdown the cache from server XXXXXXXXX6498

4. Started the cache again using ./cluster.sh

5. Exception from server XXXXXXXXX6499 is thrown as

14:34:03,239 ERROR [org.jgroups.protocols.UDP] (unicast receiver,shared=udp) failed receiving packet: java.lang.OutOfMemoryError: unable to create new native thread

  at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_17]

  at java.lang.Thread.start(Thread.java:691) [rt.jar:1.7.0_17]

  at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) [rt.jar:1.7.0_17]

  at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360) [rt.jar:1.7.0_17]

  at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1519) [jgroups-3.4.1.Final.jar:3.4.1.Final]

  at org.jgroups.protocols.TP.receive(TP.java:1448) [jgroups-3.4.1.Final.jar:3.4.1.Final]

  at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:692) [jgroups-3.4.1.Final.jar:3.4.1.Final]

  at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17]

5. Exception from server XXXXXXXXX6498 is thrown as

Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache DevelopmentLocationCache on XXXXXXXXX6498/clustered

  at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:199)

  at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source) [:1.7.0_17]

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_17]

  at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_17]

  at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)

  ... 18 more

Please note that there is no data in the cache. They are empty. Attached are the cluster.xml and logs from both the machines.

Could you please help me understand and resolve this issue.