How to deploy newer version of an application in production
I have a cluster environment consisting of two nodes on AWS with Wildfly 10. When exactly to two war on each of them, the environment works fine. But when I want to upgrade my application, I face some problems. What I want to do is;
1. Stop node1.
2. Update application on node1.
3. Start node1.
4. Node1 joins the cluster environment.
5. Stop node2.
6. Update application on node2.
7. Start node2.
8. Node2 joins the cluster environment.
But I face the problem on step 3.
My application have an infinispan replecated cache.
Infinispan cache definition on standalone-ha.xml is;
<subsystem xmlns="urn:jboss:domain:infinispan:4.0"> <cache-container name="aero" default-cache="default"> <transport lock-timeout="60000"/> <replicated-cache name="default" mode="SYNC"> <transaction mode="BATCH"/> </replicated-cache> </cache-container> ...
Infinispan cache definition on Java is;
public class ApplicationCache {
@Resource(lookup = "java:jboss/infinispan/cache/aero/default")
private Cache<String, String> cache;
.....
}
The stack trace on node1 at step3 is;
15:37:51,190 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 72) MSC000001: Failed to start service jboss.infinispan.aero.default: org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:870)
at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:639)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:628)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:531)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:222)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:849)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:621)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:572)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
at org.jboss.as.clustering.infinispan.DefaultCacheContainer.lambda$getCache$6(DefaultCacheContainer.java:119)
at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:120)
at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:114)
at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(CacheBuilder.java:80)
at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102)
... 4 more
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:224)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
... 18 more
15:37:51,253 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "aero"),
("replicated-cache" => "default")
]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
15:37:51,261 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "aero"),
("replicated-cache" => "default"),
("component" => "backup-for")
]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
15:37:51,262 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "aero"),
("replicated-cache" => "default"),
("component" => "backups")
]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
15:37:51,264 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "aero"),
("replicated-cache" => "default"),
("component" => "eviction")
]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
15:37:51,268 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "aero"),
("replicated-cache" => "default"),
("component" => "expiration")
]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
15:37:51,269 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "aero"),
("replicated-cache" => "default"),
("component" => "locking")
]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
15:37:51,270 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "aero"),
("replicated-cache" => "default"),
("component" => "partition-handling")
]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
15:37:51,270 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "aero"),
("replicated-cache" => "default"),
("component" => "state-transfer")
]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
15:37:51,271 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "aero"),
("replicated-cache" => "default"),
("store" => "none")
]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.aero.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.aero.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1"}}
The stack trace on node2 at step3 is;
2016-09-02 15:33:51,793 ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread--p20-t23) Failed to send entries to node node1 : ISPN000217: Received exception from node1, see cause for remote stack trace: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node1, see cause for remote stack trace at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:760) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.futureDone(SingleResponseFuture.java:30) at org.jgroups.blocks.Request.checkCompletion(Request.java:169) at org.jgroups.blocks.UnicastRequest.receiveResponse(UnicastRequest.java:83) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:398) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684) at org.jgroups.JChannel.up(JChannel.java:738) at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:119) at org.jgroups.stack.Protocol.up(Protocol.java:374) at org.jgroups.protocols.FORK.up(FORK.java:114) at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:394) at org.jgroups.protocols.FlowControl.up(FlowControl.java:383) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1045) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234) at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:805) at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:422) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:638) at org.jgroups.protocols.BARRIER.up(BARRIER.java:152) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155) at org.jgroups.protocols.FD.up(FD.java:260) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310) at org.jgroups.protocols.MERGE2.up(MERGE2.java:237) at org.jgroups.protocols.Discovery.up(Discovery.java:295) at org.jgroups.protocols.TP.passMessageUp(TP.java:1577) at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.infinispan.commons.CacheException: Problems invoking command. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:163) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) ... 25 more Caused by: java.io.InvalidClassException: com.altair.aquila.core.auth.entity.b; Module load failed at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:104) at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:948) at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1255) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:152) at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:142) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:475) at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:234) at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:149) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) at org.infinispan.marshall.exts.ListExternalizer.readObject(ListExternalizer.java:63) at org.infinispan.marshall.exts.ListExternalizer.readObject(ListExternalizer.java:26) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:475) at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:234) at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:149) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) at org.infinispan.statetransfer.StateChunk$Externalizer.readObject(StateChunk.java:88) at org.infinispan.statetransfer.StateChunk$Externalizer.readObject(StateChunk.java:65) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:475) at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:234) at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:149) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) at org.infinispan.marshall.exts.ListExternalizer.readObject(ListExternalizer.java:63) at org.infinispan.marshall.exts.ListExternalizer.readObject(ListExternalizer.java:26) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:475) at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:234) at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:149) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:109) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:158) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:68) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:475) at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:234) at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:149) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:134) at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101) at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80) at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28) at org.jboss.as.clustering.infinispan.ChannelTransport$1.objectFromBuffer(ChannelTransport.java:57) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:143) ... 27 more Caused by: org.jboss.modules.ModuleNotFoundException: deployment.maintage-aero-web-1.2.15.war:main at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:223) at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:102) ... 79 more 2016-09-02 15:33:51,868 INFO [org.infinispan.CLUSTER] (remote-thread--p11-t12) ISPN000336: Finished cluster-wide rebalance for cache maintage-balloon-web-1.2.15.war, topology id = 7 2016-09-02 15:34:41,566 WARN [org.wildfly.clustering.web.infinispan] (default task-13) WFLYCLWEBINF0005: Failed to cancel expiration/passivation of session RYs3tq6cPy5QqMnTLaa59LUPyvznlQCxqC_nyq33 on primary owner.: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: timeout sending message to node1 at org.jgroups.blocks.UnicastRequest.getValue(UnicastRequest.java:203) at org.jgroups.blocks.UnicastRequest.get(UnicastRequest.java:212) at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcher.executeOnNode(ChannelCommandDispatcher.java:151) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.lambda$executeOnPrimaryOwner$22(InfinispanSessionManager.java:197) at org.wildfly.clustering.ee.infinispan.RetryingInvoker.invoke(RetryingInvoker.java:70) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.executeOnPrimaryOwner(InfinispanSessionManager.java:194) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.cancel(InfinispanSessionManager.java:179) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:246) at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140) at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726) at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370) at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47) at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231) at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152) at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1$1.run(ServletInitialHandler.java:180) at java.security.AccessController.doPrivileged(Native Method) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:177) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException: timeout sending message to node1 ... 26 more 2016-09-02 15:34:41,607 WARN [org.wildfly.clustering.web.infinispan] (default task-13) WFLYCLWEBINF0006: Failed to schedule expiration/passivation of session RYs3tq6cPy5QqMnTLaa59LUPyvznlQCxqC_nyq33 on primary owner.: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: timeout sending message to node1 at org.jgroups.blocks.UnicastRequest.getValue(UnicastRequest.java:203) at org.jgroups.blocks.UnicastRequest.get(UnicastRequest.java:212) at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcher.executeOnNode(ChannelCommandDispatcher.java:151) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.lambda$executeOnPrimaryOwner$22(InfinispanSessionManager.java:197) at org.wildfly.clustering.ee.infinispan.RetryingInvoker.invoke(RetryingInvoker.java:70) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.executeOnPrimaryOwner(InfinispanSessionManager.java:194) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.schedule(InfinispanSessionManager.java:187) at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager$SchedulableSession.close(InfinispanSessionManager.java:456) at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:76) at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:768) at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:557) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:331) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1$1.run(ServletInitialHandler.java:180) at java.security.AccessController.doPrivileged(Native Method) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:177) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException: timeout sending message to node1 ... 22 more
When there are two different version on each node, session replication also does not work.
What is my problem and how can I fix it?