Wildfly application redeployment issue in Cluster => InfiniSpan

[gegu1]

Hello Everybody

I am using WildFly 9.0.2 Final.

I set up a domain of 3 machines :

1 one hosts only the domain controller (master)

2 one hosts 1 applicative node

3 one hosts 1 applicative node

I have 1 servergroup made up of 4 jvms. 2 ones on each applicative node.

I have no issue with the first deployment of the application. But when I try to redeploy the application I get many errors with Infinispan :

- ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 17.5 seconds for valid responses from any of...

- ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology ...

- org.infinispan.CacheException: Unable to invoke method public void org.infinispan.loaders.CacheLoaderManagerImpl.start() on object of type CacheLoaderManagerImpl

etc...

I was able to isolate 2 cases where this behaviour does not appear :

1) with only 2 jvms or less per servergroup (but I need 4)

2) when at least 2 applications are deployed within the servergroup => 1 is always living when the second is redeployed

When It goes well (2 apps in JVMs - 4 JVMs) :

I do not see messages of disconnection from Infinispan cache for the JVM (the other app, is still living)

I do not ever see any INFO logs for infinispan classes in server.log

When it crashes (1 app - 4 JVMs) :

I see disconnection messages, many INFO logs for infinispan classes

and then many exceptions

I think that redeployment process, disconnects JVM from Infinispan cache, makes the cluster unresponsive from the cache point of view so that infinispan cannot recover from this situation.

How can I have it worked ???

Please assist.

Thanks

[pferraro]

Can you paste the full stack trace for "org.infinispan.CacheException: Unable to invoke method public void org.infinispan.loaders.CacheLoaderManagerImpl.start() on object of type CacheLoaderManagerImpl" ?

[gegu1]

Hello Paul,

thanks for replying, I do not exactly find again this particular execption. But I have this one...

2018-01-19 14:51:17,645 WARN  [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 993) ISPN000285: Issue when retrieving cluster listeners from master:pre-master-two response was CacheNotFoundResponse

2018-01-19 14:51:18,547 WARN  [org.infinispan.partitionhandling.impl.PreferAvailabilityStrategy] (remote-thread--p15-t5) ISPN000312: Cache dist lost data because of graceful leaver master:pre-master-two

2018-01-19 14:51:18,549 INFO  [org.infinispan.CLUSTER] (remote-thread--p15-t5) ISPN000336: Finished cluster-wide rebalance for cache dist, topology id = 12

2018-01-19 14:53:17,651 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 993) MSC000001: Failed to start service jboss.infinispan.ejb.dist: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.dist: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl

at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

at org.jboss.threads.JBossThread.run(JBossThread.java:320)

Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl

at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)

at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)

at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)

at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)

at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)

at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)

at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:850)

at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:327)

at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(CacheBuilder.java:80)

at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102)

... 4 more

Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache dist on master:pre-master-one

at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:222)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)

... 13 more

I have stored only 1 log file...

Now as I have found a bypass to have it worked, I put it in place to let people work in my company. Waiting for the good solution to implement my goal...

Hope it will help

[pferraro]

The default timeout for the initial state transfer is 1 minute.  If find that this isn't sufficient, you should probably disable initial state transfer altogether.

e.g.

<state-transfer enabled="false"/>

or

/subsystem=infinispan/cache-container=foo/distributed-cache=bar/component=state-transfer:write-attribute(name=enabled, value=false){allow-resource-service-restart=true}