3 Replies Latest reply on Jan 26, 2018 8:23 AM by Paul Ferraro

    Wildfly application redeployment issue in Cluster => InfiniSpan

    Guillaume GELB Newbie

      Hello Everybody

       

      I am using WildFly 9.0.2 Final.

       

      I set up a domain of 3 machines :

      1 one hosts only the domain controller (master)

      2 one hosts 1 applicative node

      3 one hosts 1 applicative node

       

      I have 1 servergroup made up of 4 jvms. 2 ones on each applicative node.

       

      I have no issue with the first deployment of the application. But when I try to redeploy the application I get many errors with Infinispan :

      - ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for 17.5 seconds for valid responses from any of...

      - ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology ...

      - org.infinispan.CacheException: Unable to invoke method public void org.infinispan.loaders.CacheLoaderManagerImpl.start() on object of type CacheLoaderManagerImpl

      etc...

       

      I was able to isolate 2 cases where this behaviour does not appear :

      1) with only 2 jvms or less per servergroup (but I need 4)

      2) when at least 2 applications are deployed within the servergroup => 1 is always living when the second is redeployed

       

      When It goes well (2 apps in JVMs - 4 JVMs) :

      I do not see messages of disconnection from Infinispan cache for the JVM (the other app, is still living)

      I do not ever see any INFO logs for infinispan classes in server.log

       

      When it crashes (1 app - 4 JVMs) :

      I see disconnection messages, many INFO logs for infinispan classes

      and then many exceptions

       

       

      I think that redeployment process, disconnects JVM from Infinispan cache, makes the cluster unresponsive from the cache point of view so that infinispan cannot recover from this situation.

       

      How can I have it worked ???

      Please assist.

       

      Thanks

        • 1. Re: Wildfly application redeployment issue in Cluster => InfiniSpan
          Paul Ferraro Master

          Can you paste the full stack trace for "org.infinispan.CacheException: Unable to invoke method public void org.infinispan.loaders.CacheLoaderManagerImpl.start() on object of type CacheLoaderManagerImpl" ?

          • 2. Re: Wildfly application redeployment issue in Cluster => InfiniSpan
            Guillaume GELB Newbie

            Hello Paul,

            thanks for replying, I do not exactly find again this particular execption. But I have this one...

             

            2018-01-19 14:51:17,645 WARN  [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 993) ISPN000285: Issue when retrieving cluster listeners from master:pre-master-two response was CacheNotFoundResponse

            2018-01-19 14:51:18,547 WARN  [org.infinispan.partitionhandling.impl.PreferAvailabilityStrategy] (remote-thread--p15-t5) ISPN000312: Cache dist lost data because of graceful leaver master:pre-master-two

            2018-01-19 14:51:18,549 INFO  [org.infinispan.CLUSTER] (remote-thread--p15-t5) ISPN000336: Finished cluster-wide rebalance for cache dist, topology id = 12

            2018-01-19 14:53:17,651 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 993) MSC000001: Failed to start service jboss.infinispan.ejb.dist: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.dist: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl

            at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

            at java.lang.Thread.run(Thread.java:745)

            at org.jboss.threads.JBossThread.run(JBossThread.java:320)

            Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl

            at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)

            at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)

            at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)

            at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)

            at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)

            at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)

            at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:850)

            at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:327)

            at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(CacheBuilder.java:80)

            at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102)

            ... 4 more

            Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache dist on master:pre-master-one

            at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:222)

            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

            at java.lang.reflect.Method.invoke(Method.java:497)

            at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)

            ... 13 more

             

            I have stored only 1 log file...

            Now as I have found a bypass to have it worked, I put it in place to let people work in my company. Waiting for the good solution to implement my goal...

             

            Hope it will help

            • 3. Re: Wildfly application redeployment issue in Cluster => InfiniSpan
              Paul Ferraro Master

              The default timeout for the initial state transfer is 1 minute.  If find that this isn't sufficient, you should probably disable initial state transfer altogether.

              e.g.

              <state-transfer enabled="false"/>

              or

              /subsystem=infinispan/cache-container=foo/distributed-cache=bar/component=state-transfer:write-attribute(name=enabled, value=false){allow-resource-service-restart=true}