0 Replies Latest reply on Aug 1, 2017 6:39 AM by SUJIT RAJAN

    Replication timeout exception is occuring often in infinispan replicated cache added in wildfly 10 cluster(2 nodes)

    SUJIT RAJAN Newbie

      Hi All,

       

      We are setting up a clustered environment using wildfly 10 of 2 nodes. We have added a new cache-container & replicated cache named 'userSessionData' for sharing data across 2 servers.Standalone-full-ha.xml is used. Everything is working fine in our development environment. But in client pre-prod environment, quite often, we are getting replication timeout exception. In our dev environment, we are using default remote-timeout setting i.e. 17500 ms, which I think corresponds to replication timeout. In client pre-prod env. , we have increased it to 60000 ms i.e. 1 min, but still getting same exception quite often. One of the timeout issue scenario:- When user login into application,then we put user session details into cache. During writing into cache, exception occurs. I have attached infinispan configuration, java codes used during cache writing & exception.  Also pasted it below.

       

      Please help me as soon as possible.It's very urgent. Without this we can't make our prod live.

       

      Below is the infinispan configuration for added cache.

       

      <cache-container name="userSessionData_cache" default-cache="default" jndi-name="infinispan/userSessionData_cache">

                      <transport lock-timeout="60000"/>

                      <replicated-cache name="userSessionData" jndi-name="infinispan/userSessionData_cache/userSessionData" mode="SYNC" remote-timeout="60000">

                          <transaction mode="BATCH"/>

                      </replicated-cache>

      </cache-container>

       

      Below is the java class code which will call another java class used for cache:

       

      public class UserSessionTracker {

       

          private UserSessionTracker() {

          }

          private static UserSessionTracker instance = new UserSessionTracker();

       

          public static void putUserSession(String userName, String sessionId) {

              synchronized (instance) {

                      UserSessionDataCacheManager.getInstance().putUserSession(userName, sessionId);

              }

          }

       

      Below is the java class used for cache (Singleton pattern for class is used ) :

       

      @ManagedBean(name = "userSessionDataCacheManager", eager = true)

      @ApplicationScoped

      public class UserSessionDataCacheManager {

       

      @Resource(name = "java:jboss/infinispan/userSessionData_cache/userSessionData")

          private Cache<String, java.util.Hashtable<String, Object>> entityCache = null;

       

          private static UserSessionDataCacheManager configInstance = null;

       

          public UserSessionDataCacheManager() {

              configInstance = this;

          }

       

          @PostConstruct

          public void start() {

      System.out.println("Cache name: " + entityCache.getName());

          }

       

          public static UserSessionDataCacheManager getInstance() {

              if (configInstance == null) {

                  createInstance();

              }

              return configInstance;

          }

       

          private static void createInstance() {

              configInstance = new UserSessionDataCacheManager();

          }

        

          public void putUserSession(String userName, String sessionId) {

      System.out.println("inside  putUserSession");

          java.util.Hashtable<String, Object> sessionData = new java.util.Hashtable<String, Object>();

              sessionData.put("Session", sessionId);

              entityCache.put(userName, sessionData);

       

          }

      }

       

      Below is the exception log.

       

      2017-07-27 11:53:54,539 INFO  [stdout] (default task-15) inside  putUserSession

       

      2017-07-27 11:54:54,571 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-15) ISPN000136: Error executing command PrepareCommand, writing keys [F2959267]: org.infinispan.util.concurrent.TimeoutException: Replication timeout for fit-r3stems01

      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)

      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)

      at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)

      at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

      at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

      at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)

      at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)

      at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)

      at java.util.concurrent.FutureTask.run(FutureTask.java:266)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

      at java.lang.Thread.run(Thread.java:748)

       

       

      2017-07-27 11:54:54,571 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-15) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: Replication timeout for fit-r3stems01

      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)

      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)

      at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)

      at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

      at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

      at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)

      at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)

      at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)

      at java.util.concurrent.FutureTask.run(FutureTask.java:266)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

      at java.lang.Thread.run(Thread.java:748)

       

       

      2017-07-27 11:55:54,587 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-15) ISPN000136: Error executing command RollbackCommand, writing keys [F2959267]: org.infinispan.util.concurrent.TimeoutException: Replication timeout for fit-r3stems01

      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)

      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)

      at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)

      at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

      at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

      at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)

      at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)

      at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)

      at java.util.concurrent.FutureTask.run(FutureTask.java:266)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

      at java.lang.Thread.run(Thread.java:748)

       

       

      2017-07-27 11:55:54,587 WARN  [org.infinispan.transaction.impl.TransactionCoordinator] (default task-15) ISPN000141: Could not rollback prepared 1PC transaction. This transaction will be rolled back by the recovery process, if enabled. Transaction: LocalXaTransaction{xid=DummyXid{id=77}} LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=[], backupKeyLocks=[], topologyId=3, stateTransferFlag=null} org.infinispan.transaction.xa.LocalXaTransaction@8b: org.infinispan.util.concurrent.TimeoutException: Replication timeout for fit-r3stems01

      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)

      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)

      at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)

      at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

      at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

      at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)

      at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)

      at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)

      at java.util.concurrent.FutureTask.run(FutureTask.java:266)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

      at java.lang.Thread.run(Thread.java:748)

       

       

      2017-07-27 11:55:54,587 WARN  [org.infinispan.transaction.tm.DummyTransaction] (default task-15) ISPN000112: exception while committing: javax.transaction.xa.XAException

      at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:207)

      at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)

      at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:114)

      at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:401)

      at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:448)

      at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:321)

      at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:108)

      at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:71)

      at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:81)

      at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:814)

      at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:565)

      at io.undertow.servlet.spec.HttpServletResponseImpl.sendRedirect(HttpServletResponseImpl.java:202)

      at org.apache.struts2.dispatcher.ServletRedirectResult.sendRedirect(ServletRedirectResult.java:257)

      at org.apache.struts2.dispatcher.ServletRedirectResult.doExecute(ServletRedirectResult.java:229)

      at org.apache.struts2.dispatcher.StrutsResultSupport.execute(StrutsResultSupport.java:188)

      at org.apache.struts2.dispatcher.ServletRedirectResult.execute(ServletRedirectResult.java:164)

      at com.opensymphony.xwork2.DefaultActionInvocation.executeResult(DefaultActionInvocation.java:369)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:273)

      at com.fnb.common.interceptors.AuthenticationInterceptor.intercept(AuthenticationInterceptor.java:58)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.DeprecationInterceptor.intercept(DeprecationInterceptor.java:41)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:256)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:167)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:265)

      at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:76)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:138)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:191)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:73)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.DateTextFieldInterceptor.intercept(DateTextFieldInterceptor.java:125)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:91)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:253)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:100)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:141)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:145)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:139)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:193)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:189)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.DeprecationInterceptor.intercept(DeprecationInterceptor.java:41)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:138)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:73)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.DateTextFieldInterceptor.intercept(DateTextFieldInterceptor.java:125)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:91)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:189)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:167)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.json.JSONValidationInterceptor.doIntercept(JSONValidationInterceptor.java:116)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:265)

      at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:76)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.DeprecationInterceptor.intercept(DeprecationInterceptor.java:41)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:138)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:73)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.DateTextFieldInterceptor.intercept(DateTextFieldInterceptor.java:125)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:91)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:189)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:54)

      at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:564)

      at org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:81)

      at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:99)

      at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

      at com.fnb.common.filters.ParamHandlerFilter.doFilter(ParamHandlerFilter.java:35)

      at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

      at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

      at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

      at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

      at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

      at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

      at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

      at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

      at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

      at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

      at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)

      at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)

      at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)

      at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)

      at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)

      at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)

      at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

      at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)

      at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

      at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

      at java.lang.Thread.run(Thread.java:748)

      Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for fit-r3stems01

      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)

      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)

      at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)

      at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

      at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

      at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)

      at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)

      at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)

      at java.util.concurrent.FutureTask.run(FutureTask.java:266)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

      ... 3 more

       

       

      2017-07-27 11:55:54,712 WARN  [org.wildfly.clustering.web.undertow] (default task-15) javax.transaction.HeuristicMixedException: org.infinispan.commons.CacheException: javax.transaction.HeuristicMixedException

      at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:74)

      at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:81)

      at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:814)

      at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:565)

      at io.undertow.servlet.spec.HttpServletResponseImpl.sendRedirect(HttpServletResponseImpl.java:202)

      at org.apache.struts2.dispatcher.ServletRedirectResult.sendRedirect(ServletRedirectResult.java:257)

      at org.apache.struts2.dispatcher.ServletRedirectResult.doExecute(ServletRedirectResult.java:229)

      at org.apache.struts2.dispatcher.StrutsResultSupport.execute(StrutsResultSupport.java:188)

      at org.apache.struts2.dispatcher.ServletRedirectResult.execute(ServletRedirectResult.java:164)

      at com.opensymphony.xwork2.DefaultActionInvocation.executeResult(DefaultActionInvocation.java:369)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:273)

      at com.fnb.common.interceptors.AuthenticationInterceptor.intercept(AuthenticationInterceptor.java:58)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.DeprecationInterceptor.intercept(DeprecationInterceptor.java:41)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:256)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:167)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:265)

      at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:76)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:138)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:191)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:73)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.DateTextFieldInterceptor.intercept(DateTextFieldInterceptor.java:125)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:91)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:253)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:100)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:141)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:145)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:139)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:193)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:189)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.DeprecationInterceptor.intercept(DeprecationInterceptor.java:41)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:138)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:73)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.DateTextFieldInterceptor.intercept(DateTextFieldInterceptor.java:125)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:91)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:189)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:167)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.json.JSONValidationInterceptor.doIntercept(JSONValidationInterceptor.java:116)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:265)

      at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:76)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.DeprecationInterceptor.intercept(DeprecationInterceptor.java:41)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:138)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:73)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.DateTextFieldInterceptor.intercept(DateTextFieldInterceptor.java:125)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:91)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171)

      at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:189)

      at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244)

      at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:54)

      at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:564)

      at org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:81)

      at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:99)

      at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

      at com.fnb.common.filters.ParamHandlerFilter.doFilter(ParamHandlerFilter.java:35)

      at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

      at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

      at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

      at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

      at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

      at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

      at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

      at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

      at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

      at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

      at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)

      at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)

      at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)

      at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)

      at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)

      at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)

      at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

      at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)

      at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

      at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

      at java.lang.Thread.run(Thread.java:748)

      Caused by: javax.transaction.HeuristicMixedException

      at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:439)

      at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:448)

      at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:321)

      at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:108)

      at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:71)

      ... 161 more

      Caused by: javax.transaction.xa.XAException

      at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:207)

      at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)

      at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:114)

      at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:401)

      ... 165 more

      Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for fit-r3stems01

      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)

      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)

      at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)

      at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

      at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

      at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)

      at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)

      at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)

      at java.util.concurrent.FutureTask.run(FutureTask.java:266)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

      ... 3 more