Infinispan 8.2.4.Final - TimeoutException: ISPN000299: Unable to acquire lock after 2 seconds for key & ISPN000136: Error executing command PutKeyValueCommand, writing keys
zeguan Sep 21, 2017 11:57 AMI have 3 tomcat under 3 different VMs, the web app under each tomcat is using Infinispan 8.2.4.Final as embedded mode, distributed cache, owners = 3, by putAsync
I got 3 kinds of exception when put item volume grew
1) org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 2 seconds for key
I can see the key had been locked by another requestor, as the key had been put by putAsync, it should be handled by thread pool instead of current thread
from exception trace, ‘ThreadPoolExecutor.execute’ , I guess the capacity of theadpool has been reached, so it got rejected, then current thread continue handling the put command, waiting for other requestor releasing the lock but timeout (2s),
is my understanding correct?
2) ISPN000136: Error executing command PutKeyValueCommand, writing keys
putAsync by thread: http-nio-8081-exec-19368
scheduled by thread: http-nio-8081-exec-19373
timeout and output by thread: http-nio-8081-exec-19373
I may collect the incorrect putAsync thread, but I am confused by 'scheduled thread and current thead are sync instead of aysnc '
3) ISPN000136: Error executing command PutKeyValueCommand, writing keys
putAsync by thread: http-nio-8081-exec-19245
scheduled by thread: http-nio-8081-exec-19245
timeout and output by thread: http-nio-8081-exec-19245
almost same with 2), but putAsync thread is same with schedule and timeout output thread
regarding 2) and 3), what is the scenario that scheduling pool used request(current) thread but thread pool?
regarding those TimeoutException for growing volume, which conf item I can change to avoid it?
exception type 1)
----------------------------------------------------------------------------------------------------------------------------------------
[2017-09-16_19:16:25:781] - [00016171] [DEBUG] - com.test.it.common.framework.service.cache.CacheManagerService.put(CacheManagerService.java:82)http-nio-8081-exec-19369 - |cacheName|idatasaas|key|app_000000000414&_&pro&_&app_0000000000011111&_&-945774822
[2017-09-16_19:16:25:781] - [00016171] [DEBUG] - com.test.engine.distributedcache.impl.AppDistributedCache.put(AppDistributedCache.java:390)http-nio-8081-exec-19369 - [put] cacheName = idatasaas, key = app_000000000414&_&pro&_&app_0000000000011111&_&-945774822
[2017-09-16_19:16:27:783] - [00016171] [ERROR] - org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:140)http-nio-8081-exec-19369 - ISPN000136: Error executing command PutKeyValueCommand, writing keys [app_000000000414&_&pro&_&app_0000000000011111&_&-945774822]
org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 2 seconds for key app_000000000414&_&pro&_&app_0000000000011111&_&-945774822 and requestor CommandUUID{address=localhost-20832, id=1281042}. Lock is held by CommandUUID{address=localhost-20832, id=1279322}
at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:238) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:193) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:98) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:41) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:65) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:349) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:281) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:107) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:191) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:177) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:83) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:78) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1672) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1121) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.cache.impl.CacheImpl.access$100(CacheImpl.java:128) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.cache.impl.CacheImpl$2.call(CacheImpl.java:1280) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:2022) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) [?:1.8.0_91]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) [?:1.8.0_91]
at org.infinispan.executors.LazyInitializingExecutorService.submit(LazyInitializingExecutorService.java:79) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.cache.impl.CacheImpl.putAsync(CacheImpl.java:1275) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.cache.impl.CacheImpl.putAsync(CacheImpl.java:1775) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.cache.impl.CacheImpl.putAsync(CacheImpl.java:293) [infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at com.test.engine.distributedcache.impl.AppDistributedCache.put(AppDistributedCache.java:452) [DistributeCache-Core-1.5.0.05.RELEASE.jar:?]
at com.test.engine.distributedcache.impl.AppDistributedCache.put(AppDistributedCache.java:395) [DistributeCache-Core-1.5.0.05.RELEASE.jar:?]
at com.test.engine.distributedcache.impl.AppDistributedCacheImpl.put(AppDistributedCacheImpl.java:55) [DistributeCache-Core-1.5.0.05.RELEASE.jar:?]
at com.test.it.common.framework.service.cache.impl.DistributedLocalCacheProvider.put(DistributedLocalCacheProvider.java:123) [FrameworkService-Release_1.1.68.5.jar:Release_1.1.68.5]
at com.test.it.common.framework.service.cache.impl.DistributedCacheProvider.put(DistributedCacheProvider.java:53) [FrameworkService-Release_1.1.68.5.jar:Release_1.1.68.5]
at com.test.it.common.framework.service.cache.CacheManagerService.put(CacheManagerService.java:83) [FrameworkService-Release_1.1.68.5.jar:Release_1.1.68.5]
at com.test.it.common.saas.idata.util.CacheManger.put(CacheManger.java:49) [iDataSaasBase-1.1.2.jar:?]
at com.test.it.common.saas.idata.biz.W3AccountService.getW3Account(W3AccountService.java:130) [iDataSaasMHRR002Core-1.0.00.jar:?]
at com.test.it.common.saas.idata.biz.PersonBatchService.getBatchPersonInfo(PersonBatchService.java:103) [iDataSaasMHRR002Core-1.0.00.jar:?]
at sun.reflect.GeneratedMethodAccessor477.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_91]
at com.test.it.common.saas.idata.util.BizCommonUtil.callServiceByMethodName(BizCommonUtil.java:217) [iDataSaasBase-1.1.2.jar:?]
at com.test.it.common.saas.idata.util.BizCommonUtil.defaultServiceInvoke(BizCommonUtil.java:182) [iDataSaasBase-1.1.2.jar:?]
at com.test.it.common.saas.idatasaasmhrr002.service.iDataSaasMHRR002Service.execute(iDataSaasMHRR002Service.java:73) [iDataSaasMHRR002Core-1.0.00.jar:?]
at com.test.it.common.framework.service.impl.CommonWSImpl.invoke(CommonWSImpl.java:411) [FrameworkService-Release_1.1.68.5.jar:Release_1.1.68.5]
at com.test.it.common.framework.service.impl.CommonWSImpl.execute(CommonWSImpl.java:178) [FrameworkService-Release_1.1.68.5.jar:Release_1.1.68.5]
at com.test.it.common.saas.idatasaasmhrr002.impl.iDataSaasMHRR002.executeTokenNoTransNoEncrypt(iDataSaasMHRR002.java:64) [classes/:?]
at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_91]
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) [cxf-core-3.1.4.jar:3.1.4]
at org.apache.cxf.jaxws.JAXWSMethodInvoker.performInvocation(JAXWSMethodInvoker.java:66) [cxf-rt-frontend-jaxws-3.1.4.jar:3.1.4]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) [cxf-core-3.1.4.jar:3.1.4]
at org.apache.cxf.jaxws.AbstractJAXWSMethodInvoker.invoke(AbstractJAXWSMethodInvoker.java:232) [cxf-rt-frontend-jaxws-3.1.4.jar:3.1.4]
at org.apache.cxf.jaxws.JAXWSMethodInvoker.invoke(JAXWSMethodInvoker.java:85) [cxf-rt-frontend-jaxws-3.1.4.jar:3.1.4]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:74) [cxf-core-3.1.4.jar:3.1.4]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) [cxf-core-3.1.4.jar:3.1.4]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_91]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$2.run(ServiceInvokerInterceptor.java:126) [cxf-core-3.1.4.jar:3.1.4]
at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37) [cxf-core-3.1.4.jar:3.1.4]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:131) [cxf-core-3.1.4.jar:3.1.4]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) [cxf-core-3.1.4.jar:3.1.4]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-core-3.1.4.jar:3.1.4]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:251) [cxf-rt-transports-http-3.1.4.jar:3.1.4]
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) [cxf-rt-transports-http-3.1.4.jar:3.1.4]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) [cxf-rt-transports-http-3.1.4.jar:3.1.4]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) [cxf-rt-transports-http-3.1.4.jar:3.1.4]
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180) [cxf-rt-transports-http-3.1.4.jar:3.1.4]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:293) [cxf-rt-transports-http-3.1.4.jar:3.1.4]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:212) [cxf-rt-transports-http-3.1.4.jar:3.1.4]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) [servlet-api.jar:?]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:268) [cxf-rt-transports-http-3.1.4.jar:3.1.4]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292) [catalina.jar:8.0.35]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.35]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.0.35]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.35]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.35]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) [catalina.jar:8.0.35]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [catalina.jar:8.0.35]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [catalina.jar:8.0.35]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [catalina.jar:8.0.35]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [catalina.jar:8.0.35]
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616) [catalina.jar:8.0.35]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [catalina.jar:8.0.35]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528) [catalina.jar:8.0.35]
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1099) [tomcat-coyote.jar:8.0.35]
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672) [tomcat-coyote.jar:8.0.35]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520) [tomcat-coyote.jar:8.0.35]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476) [tomcat-coyote.jar:8.0.35]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.0.35]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
[2017-09-16_19:16:27:793] - [00016171] [DEBUG] - com.test.engine.distributedcache.impl.AppDistributedCache.put(AppDistributedCache.java:453)http-nio-8081-exec-19369 - NOT USING_COMPRESS, key = app_000000000414&_&pro&_&app_0000000000011111&_&-945774822 ,Async, cost time:2012 ms
exception type 2)
----------------------------------------------------------------------------------------------------------------------------------------
[2017-09-16_19:16:23:863] - [00016170] [DEBUG] - com.test.engine.distributedcache.impl.AppDistributedCache.put(AppDistributedCache.java:390)http-nio-8081-exec-19368 - [put] cacheName = idatasaas, key = test.paas.portal&_&pro&_&app_0000000000011111&_&922309117
[2017-09-16_19:16:37:434] - [00016175] [ERROR] - org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:140)http-nio-8081-exec-19373 - ISPN000136: Error executing command PutKeyValueCommand, writing keys [test.paas.portal&_&pro&_&app_0000000000011111&_&922309117]
org.infinispan.util.concurrent.TimeoutException: Replication timeout for dggmwc6tmc035-48913
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:629) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_91]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[?:1.8.0_91]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_91]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[?:1.8.0_91]
at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
[2017-09-16_19:16:37:442] - [00016175] [DEBUG] - com.test.engine.distributedcache.impl.AppDistributedCache.put(AppDistributedCache.java:453)http-nio-8081-exec-19373 - NOT USING_COMPRESS, key = test.paas.portal&_&pro&_&app_0000000000011111&_&922309117 ,Async, cost time:15009 ms
exception type 3)
----------------------------------------------------------------------------------------------------------------------------------------
[2017-09-16_19:16:03:944] - [000160e0] [DEBUG] - com.test.engine.distributedcache.impl.AppDistributedCache.put(AppDistributedCache.java:390)http-nio-8081-exec-19245 - [put] cacheName = idatasaas, key = app_000000000414&_&pro&_&app_0000000000011111&_&1863878812
[2017-09-16_19:16:18:946] - [000160e0] [ERROR] - org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:140)http-nio-8081-exec-19245 - ISPN000136: Error executing command PutKeyValueCommand, writing keys [app_000000000414&_&pro&_&app_0000000000011111&_&1863878812]
org.infinispan.util.concurrent.TimeoutException: Replication timeout for dggmwc6tmc035-48913
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:629) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_91]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[?:1.8.0_91]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_91]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[?:1.8.0_91]
at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) ~[infinispan-embedded-8.2.4.Final.jar:8.2.4.Final]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
[2017-09-16_19:16:18:955] - [000160e0] [DEBUG] - com.test.engine.distributedcache.impl.AppDistributedCache.put(AppDistributedCache.java:453)http-nio-8081-exec-19245 - NOT USING_COMPRESS, key = app_000000000414&_&pro&_&app_0000000000011111&_&1863878812 ,Async, cost time:15010 ms