0 Replies Latest reply on Sep 7, 2016 3:01 AM by Ali Hopyar

    How to deploy newer version of an application in production

    Ali Hopyar Newbie

      I have a cluster environment consisting of two nodes on AWS with Wildfly 10. When exactly to two war on each of them, the environment works fine. But when I want to upgrade my application, I face some problems. What I want to do is;

       

      1. Stop node1.

      2. Update application on node1.

      3. Start node1.

      4. Node1 joins the cluster environment.

      5. Stop node2.

      6. Update application on node2.

      7. Start node2.

      8. Node2 joins the cluster environment.

       

      But I face the problem on step 3.

       

      My application have an infinispan replecated cache.

       

      Infinispan cache definition on standalone-ha.xml is;

       

       

      <subsystem xmlns="urn:jboss:domain:infinispan:4.0">  
        <cache-container name="aero" default-cache="default">  
        <transport lock-timeout="60000"/>  
        <replicated-cache name="default" mode="SYNC">  
        <transaction mode="BATCH"/>  
        </replicated-cache>  
        </cache-container>  
      ...  

       

       

      Infinispan cache definition on Java is;

       

       

      public class ApplicationCache {
      
        @Resource(lookup = "java:jboss/infinispan/cache/aero/default")
        private Cache<String, String> cache; 
      
        .....
      }

       

       

      The stack trace on node1 at step3 is;

       

      15:37:51,190 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 72) MSC000001: Failed to start service jboss.infinispan.aero.default: org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at org.jboss.threads.JBossThread.run(JBossThread.java:320)
      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
        at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:870)
        at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:639)
        at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:628)
        at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:531)
        at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:222)
        at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:849)
        at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:621)
        at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:572)
        at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
        at org.jboss.as.clustering.infinispan.DefaultCacheContainer.lambda$getCache$6(DefaultCacheContainer.java:119)
        at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:120)
        at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:114)
        at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(CacheBuilder.java:80)
        at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102)
        ... 4 more
      Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1
        at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:224)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
        ... 18 more
      
      15:37:51,253 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
        ("subsystem" => "infinispan"),
        ("cache-container" => "aero"),
        ("replicated-cache" => "default")
      ]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
      15:37:51,261 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
        ("subsystem" => "infinispan"),
        ("cache-container" => "aero"),
        ("replicated-cache" => "default"),
        ("component" => "backup-for")
      ]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
      15:37:51,262 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
        ("subsystem" => "infinispan"),
        ("cache-container" => "aero"),
        ("replicated-cache" => "default"),
        ("component" => "backups")
      ]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
      15:37:51,264 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
        ("subsystem" => "infinispan"),
        ("cache-container" => "aero"),
        ("replicated-cache" => "default"),
        ("component" => "eviction")
      ]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
      15:37:51,268 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
        ("subsystem" => "infinispan"),
        ("cache-container" => "aero"),
        ("replicated-cache" => "default"),
        ("component" => "expiration")
      ]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
      15:37:51,269 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
        ("subsystem" => "infinispan"),
        ("cache-container" => "aero"),
        ("replicated-cache" => "default"),
        ("component" => "locking")
      ]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
      15:37:51,270 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
        ("subsystem" => "infinispan"),
        ("cache-container" => "aero"),
        ("replicated-cache" => "default"),
        ("component" => "partition-handling")
      ]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
      15:37:51,270 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
        ("subsystem" => "infinispan"),
        ("cache-container" => "aero"),
        ("replicated-cache" => "default"),
        ("component" => "state-transfer")
      ]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
      15:37:51,271 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
        ("subsystem" => "infinispan"),
        ("cache-container" => "aero"),
        ("replicated-cache" => "default"),
        ("store" => "none")
      ]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
        Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}

       

      The stack trace on node2 at step3 is;

       

      2016-09-02 15:33:51,793 ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread--p20-t23) Failed to send entries to node node1 : ISPN000217: Received exception from node1, see cause for remote stack trace: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node1, see cause for remote stack trace
          at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:760)
          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599)
          at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
          at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
          at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
          at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
          at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.futureDone(SingleResponseFuture.java:30)
          at org.jgroups.blocks.Request.checkCompletion(Request.java:169)
          at org.jgroups.blocks.UnicastRequest.receiveResponse(UnicastRequest.java:83)
          at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:398)
          at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684)
          at org.jgroups.JChannel.up(JChannel.java:738)
          at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:119)
          at org.jgroups.stack.Protocol.up(Protocol.java:374)
          at org.jgroups.protocols.FORK.up(FORK.java:114)
          at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
          at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
          at org.jgroups.protocols.FlowControl.up(FlowControl.java:383)
          at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1045)
          at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
          at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:805)
          at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:422)
          at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:638)
          at org.jgroups.protocols.BARRIER.up(BARRIER.java:152)
          at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
          at org.jgroups.protocols.FD.up(FD.java:260)
          at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
          at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
          at org.jgroups.protocols.Discovery.up(Discovery.java:295)
          at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
          at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      Caused by: org.infinispan.commons.CacheException: Problems invoking command.
          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:163)
          at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
          at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
          ... 25 more
      Caused by: java.io.InvalidClassException: com.altair.aquila.core.auth.entity.b; Module load failed
          at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:104)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:948)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1255)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
          at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:152)
          at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:142)
          at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:475)
          at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:234)
          at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:149)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
          at org.infinispan.marshall.exts.ListExternalizer.readObject(ListExternalizer.java:63)
          at org.infinispan.marshall.exts.ListExternalizer.readObject(ListExternalizer.java:26)
          at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:475)
          at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:234)
          at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:149)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
          at org.infinispan.statetransfer.StateChunk$Externalizer.readObject(StateChunk.java:88)
          at org.infinispan.statetransfer.StateChunk$Externalizer.readObject(StateChunk.java:65)
          at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:475)
          at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:234)
          at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:149)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
          at org.infinispan.marshall.exts.ListExternalizer.readObject(ListExternalizer.java:63)
          at org.infinispan.marshall.exts.ListExternalizer.readObject(ListExternalizer.java:26)
          at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:475)
          at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:234)
          at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:149)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
          at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:109)
          at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:158)
          at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:68)
          at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:475)
          at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:234)
          at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:149)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
          at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:134)
          at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
          at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
          at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
          at org.jboss.as.clustering.infinispan.ChannelTransport$1.objectFromBuffer(ChannelTransport.java:57)
          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:143)
          ... 27 more
      Caused by: org.jboss.modules.ModuleNotFoundException: deployment.maintage-aero-web-1.2.15.war:main
          at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:223)
          at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:102)
          ... 79 more
      
      
      2016-09-02 15:33:51,868 INFO  [org.infinispan.CLUSTER] (remote-thread--p11-t12) ISPN000336: Finished cluster-wide rebalance for cache maintage-balloon-web-1.2.15.war, topology id = 7
      2016-09-02 15:34:41,566 WARN  [org.wildfly.clustering.web.infinispan] (default task-13) WFLYCLWEBINF0005: Failed to cancel expiration/passivation of session RYs3tq6cPy5QqMnTLaa59LUPyvznlQCxqC_nyq33 on primary owner.: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: timeout sending message to node1
          at org.jgroups.blocks.UnicastRequest.getValue(UnicastRequest.java:203)
          at org.jgroups.blocks.UnicastRequest.get(UnicastRequest.java:212)
          at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcher.executeOnNode(ChannelCommandDispatcher.java:151)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.lambda$executeOnPrimaryOwner$22(InfinispanSessionManager.java:197)
          at org.wildfly.clustering.ee.infinispan.RetryingInvoker.invoke(RetryingInvoker.java:70)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.executeOnPrimaryOwner(InfinispanSessionManager.java:194)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.cancel(InfinispanSessionManager.java:179)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:246)
          at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)
          at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726)
          at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)
          at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
          at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231)
          at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
          at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245)
          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283)
          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
          at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
          at io.undertow.servlet.handlers.ServletInitialHandler$1$1.run(ServletInitialHandler.java:180)
          at java.security.AccessController.doPrivileged(Native Method)
          at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:177)
          at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      Caused by: java.util.concurrent.TimeoutException: timeout sending message to node1
          ... 26 more
      
      
      2016-09-02 15:34:41,607 WARN  [org.wildfly.clustering.web.infinispan] (default task-13) WFLYCLWEBINF0006: Failed to schedule expiration/passivation of session RYs3tq6cPy5QqMnTLaa59LUPyvznlQCxqC_nyq33 on primary owner.: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: timeout sending message to node1
          at org.jgroups.blocks.UnicastRequest.getValue(UnicastRequest.java:203)
          at org.jgroups.blocks.UnicastRequest.get(UnicastRequest.java:212)
          at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcher.executeOnNode(ChannelCommandDispatcher.java:151)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.lambda$executeOnPrimaryOwner$22(InfinispanSessionManager.java:197)
          at org.wildfly.clustering.ee.infinispan.RetryingInvoker.invoke(RetryingInvoker.java:70)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.executeOnPrimaryOwner(InfinispanSessionManager.java:194)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.schedule(InfinispanSessionManager.java:187)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager$SchedulableSession.close(InfinispanSessionManager.java:456)
          at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:76)
          at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:768)
          at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:557)
          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:331)
          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
          at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
          at io.undertow.servlet.handlers.ServletInitialHandler$1$1.run(ServletInitialHandler.java:180)
          at java.security.AccessController.doPrivileged(Native Method)
          at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:177)
          at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      Caused by: java.util.concurrent.TimeoutException: timeout sending message to node1
          ... 22 more
      

       

       

      When there are two different version on each node, session replication also does not work.

       

      What is my problem and how can I fix it?