Problem with getCache() on reconnecting nodes

[ntsankov]

Hi,

I have three node cluster setup, which is working quite fine, until I shutdown one of the nodes and then start it again. Then I usually get stuck at cacheManager.getCache() (sometimes it goes well and reconnects ok). Infinispan is 4.1.0.CR3, jgroups-2.10.0.GA, using the example tcp configuration with tcpping instead of mping. I also noticed that if I shutdown one of the working nodes, the one that was waiting got woken and data was replicated to it.

Here is the stacktrace:

{code} 
"main" prio=5 tid=0x0000000801858000 nid=0x800a08e40 waiting on condition [0x00007fffffbfd000..0x00007fffffbfed10]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000008275e54f0> (a java.util.concurrent.FutureTask$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.infinispan.distribution.DistributionManagerImpl.waitForJoinToComplete(DistributionManagerImpl.java:153)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:170)
        at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:852)
        at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:672)
        at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:574)
        at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:148)
        at org.infinispan.CacheDelegate.start(CacheDelegate.java:305)
        at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:448)
        at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:411)
        at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:388)
        at com.test.infinispan.CacheServer.getCache(CacheServer.java:60)
        - locked <0x00000008054f8700> (a java.lang.Class for com.test.infinispan.CacheServer)
        at com.test.infinispan.CacheServer.main(CacheServer.java:113)
{code}

Here is the code used to start the cache, nothing special

{code}
EmbeddedCacheManager cacheManager = new DefaultCacheManager("clustered.xml");
cache = cacheManager.getCache();//this call blocks
cache.addListener(new MyListener());
{code}

[galder.zamarreno]

The join is waiting for rehashing or rebalancing of keys after the new cache started up. Rehashing is happening in a thread whose name starts with "Rehasher-". You should check the thread dump to see what that rehasher thread is doing. The main thread is waiting for this thread to finish.