Getting lots of exception related to infinispan...| JBoss.org Content Archive (Read Only)

30. Re: Getting lots of exception related to infinispan while load testing

olegsuknov Jul 14, 2014 8:55 AM (in response to rishikesh_darandale)

Hi Paul!

I have a problem with cache replication as you can see in quotation below.
There are no application data in session but there are some exceptions after wildfly attempted removing session's entry(ies) from the cache.

wildfly-8.1.0-Final

2 servers in cluster

2014-06-30 15:57:15,166 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-60) ISPN000136: Execution error: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.lang.NullPointerException] while invoking method [public void org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.removed(org.infinispan.notifications.cachelistener.event.CacheEntryRemovedEvent)] on listener instance: org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager@1bd3806d

at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:211) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation.invoke(AbstractListenerImpl.java:229) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation.invoke(AbstractListenerImpl.java:192) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryRemoved(CacheNotifierImpl.java:230) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.notify(RemoveCommand.java:96) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.performRemove(RemoveCommand.java:213) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.perform(RemoveCommand.java:92) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:91) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.distribution.TxDistributionInterceptor.handleTxWriteCommand(TxDistributionInterceptor.java:275) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRemoveCommand(TxDistributionInterceptor.java:80) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.CacheLoaderInterceptor.visitRemoveCommand(CacheLoaderInterceptor.java:132) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:321) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:402) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.EntryWrappingInterceptor.visitRemoveCommand(EntryWrappingInterceptor.java:216) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitRemoveCommand(OptimisticLockingInterceptor.java:135) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:255) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.TxInterceptor.visitRemoveCommand(TxInterceptor.java:196) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:206) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:156) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:166) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:79) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1306) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:407) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.CacheImpl.remove(CacheImpl.java:400) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.DecoratedCache.remove(DecoratedCache.java:406) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:281) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.jboss.as.clustering.infinispan.invoker.Remover$RemoveOperation.invoke(Remover.java:49)

at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)

at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:82)

at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.remove(CoarseSessionFactory.java:117)

at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.remove(CoarseSessionFactory.java:55)

at org.wildfly.clustering.web.infinispan.session.InfinispanSession.invalidate(InfinispanSession.java:74)

at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:157)

at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:115)

at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:677) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:353) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]

at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:168) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]

at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:153) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]

at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:216) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:239) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]

at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

Caused by: java.lang.NullPointerException

at org.wildfly.clustering.web.infinispan.session.coarse.CoarseImmutableSessionAttributes.getAttributeNames(CoarseImmutableSessionAttributes.java:51)

at org.wildfly.clustering.web.session.ImmutableHttpSessionAdapter.getAttributeNames(ImmutableHttpSessionAdapter.java:80)

at org.jboss.weld.context.beanstore.http.AbstractSessionBeanStore.getAttributeNames(AbstractSessionBeanStore.java:48) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]

at org.jboss.weld.context.beanstore.AttributeBeanStore.getPrefixedAttributeNames(AttributeBeanStore.java:198) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]

at org.jboss.weld.context.beanstore.AttributeBeanStore.attach(AttributeBeanStore.java:95) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]

at org.jboss.weld.context.AbstractBoundContext.activate(AbstractBoundContext.java:66) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]

at org.jboss.weld.servlet.WeldTerminalListener.sessionDestroyed(WeldTerminalListener.java:60) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]

at io.undertow.servlet.core.ApplicationListeners.sessionDestroyed(ApplicationListeners.java:264) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at org.wildfly.clustering.web.undertow.session.UndertowSessionContext$1.sessionDestroyed(UndertowSessionContext.java:60)

at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.removed(InfinispanSessionManager.java:236)

at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source) [:1.7.0_51]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]

at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]

at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:207) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

... 86 more

2014-06-30 15:57:15,181 ERROR [io.undertow.request] (default task-60) UT005023: Exception handling request to /hl/hl.jsp: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.lang.NullPointerException] while invoking method [public void org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.removed(org.infinispan.notifications.cachelistener.event.CacheEntryRemovedEvent)] on listener instance: org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager@1bd3806d

at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:211) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

domain.xml:

...
<subsystem xmlns="urn:jboss:domain:infinispan:2.0">
...
<cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

<distributed-cache name="dist" start="EAGER" batching="true" mode="SYNC" remote-timeout="300000" owners="2" l1-lifespan="0">

<file-store shared="true" preload="true"/>

</distributed-cache>

</cache-container>

...

<cache-container name="ECEventsCacheContainer" default-cache="users" jndi-name="java:/infinispan/container/ECEventsCacheContainer">

<replicated-cache name="users" batching="true" mode="SYNC">

</replicated-cache>

<replicated-cache name="operators" batching="true" mode="SYNC">

</replicated-cache>

</cache-container>

</subsystem>

...

...
<stack name="tcp">

300

</property>

20000

</property>

</transport>

</protocol>

false

</property>

false

</property>

</protocol>

</stack>

</subsystem>

31. Re: Getting lots of exception related to infinispan while load testing

ctomc Jul 14, 2014 11:36 AM (in response to olegsuknov)

NPE almost always indicates a bug.

can you create bugreport for this in WildFly - JBoss Issue Tracker

32. Re: Getting lots of exception related to infinispan while load testing

olegsuknov Jul 15, 2014 3:25 AM (in response to ctomc)

[WFLY-3628] removing session's entry(ies) from the cache error - JBoss Issue Tracker

33. Re: Getting lots of exception related to infinispan while load testing

pferraro Jul 21, 2014 3:38 PM (in response to olegsuknov)

I'm almost certain this NPE was fixed via WFLY-3345. Can you verify?

34. Re: Getting lots of exception related to infinispan while load testing

rishikesh_darandale Aug 1, 2014 9:36 AM (in response to rishikesh_darandale)

Hi Paul,

We have updated to WF 8.1.0.final. Currently we have 6 nodes in cluster(with 3 nodes in one server group).

Below mentioned is infinispan configuration and max-active-sessions are set to 500.

                <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

                    <transport cluster="${jboss.cluster.group.name}" lock-timeout="300000"/>

                    <distributed-cache name="dist" batching="true" mode="SYNC" remote-timeout="300000" owners="4" l1-lifespan="0">

                        <state-transfer timeout="300000"/>

                        <file-store/>

                    </distributed-cache>

                </cache-container>

We are observing below exceptions:

1)

06:37:16,357 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (SessionExpirationScheduler - 1) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [15 seconds] on key [RgEJ39HVpUbIWyNqJmW_8DdV] for requestor [GlobalTransaction:<p1_dc:node2/wildlfy-cluster>:53157:local]! Lock held by [GlobalTransaction:<p1_dc:node2/wildlfy-cluster>:22190:local]

[Server:node2]     at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:198)

[Server:node2]     at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:171)

[Server:node2]     at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:169)

[Server:node2]     at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitGetKeyValueCommand(PessimisticLockingInterceptor.java:70)

[Server:node2]     at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:node2]     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:node2]     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

[Server:node2]     at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

[Server:node2]     at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:node2]     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:node2]     at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:226)

[Server:node2]     at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:221)

[Server:node2]     at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:node2]     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:node2]     at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

[Server:node2]     at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

[Server:node2]     at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:node2]     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:node2]     at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:263)

[Server:node2]     at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:247)

[Server:node2]     at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

[Server:node2]     at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:node2]     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:node2]     at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:92)

[Server:node2]     at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:node2]     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:node2]     at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)

[Server:node2]     at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)

[Server:node2]     at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

[Server:node2]     at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:node2]     at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:node2]     at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:79)

[Server:node2]     at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

[Server:node2]     at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:node2]     at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)

[Server:node2]     at org.infinispan.CacheImpl.get(CacheImpl.java:377)

[Server:node2]     at org.infinispan.DecoratedCache.get(DecoratedCache.java:396)

[Server:node2]     at org.infinispan.AbstractDelegatingCache.get(AbstractDelegatingCache.java:271)

[Server:node2]     at org.jboss.as.clustering.infinispan.invoker.Locator$FindOperation.invoke(Locator.java:54)

[Server:node2]     at org.jboss.as.clustering.infinispan.invoker.Locator$LockingFindOperation.invoke(Locator.java:71)

[Server:node2]     at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)

[Server:node2]     at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:82)

[Server:node2]     at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:109)

[Server:node2]     at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:55)

[Server:node2]     at org.wildfly.clustering.web.infinispan.session.ExpiredSessionRemover.remove(ExpiredSessionRemover.java:42)

[Server:node2]     at org.wildfly.clustering.web.infinispan.session.ExpiredSessionRemover.remove(ExpiredSessionRemover.java:32)

[Server:node2]     at org.wildfly.clustering.web.infinispan.session.SessionExpirationScheduler$ExpirationTask.run(SessionExpirationScheduler.java:111)

[Server:node2]     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51]

[Server:node2]     at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]

[Server:node2]     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) [rt.jar:1.7.0_51]

[Server:node2]     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) [rt.jar:1.7.0_51]

[Server:node2]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]

[Server:node2]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]

[Server:node2]     at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

[Server:node2]     at org.jboss.threads.JBossThread.run(JBossThread.java:122)

[Server:node2]

[Server:node2] 06:37:16,372 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (SessionExpirationScheduler - 1) ISPN000136: Execution error: java.lang.IllegalStateException: Transaction DummyTransaction{xid=DummyXid{id=59279}, status=1} is not in a valid state to be invoking cache operations on.

[Server:node2]     at org.infinispan.interceptors.TxInterceptor.enlist(TxInterceptor.java:275)

[Server:node2]     at org.infinispan.interceptors.TxInterceptor.enlistIfNeeded(TxInterceptor.java:231)

[Server:node2]     at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:225)

[Server:node2]     at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:221)

[Server:node2]     at org.infinispan.commands.read..

2)

[Server:node1] ^[[31m^[[0m^[[31m07:50:52,285 ERROR [io.undertow.request] (default task-14) UT005023: Exception handling request to /myapp/Model-F2003RW3/3083/0142000: java.lang.IllegalStateException: Transaction DummyTransaction{xid=DummyXid{id=6944}, status=1} is not in a valid state to be invoking cache operations on.^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.enlist(TxInterceptor.java:275)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.enlistIfNeeded(TxInterceptor.java:231)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:225)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:221)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:263)^[[0m

[Server:node1] ^[[31m    at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:247)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:92)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:79)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)^[[0m

[Server:node1] ^[[31m    at org.infinispan.CacheImpl.get(CacheImpl.java:377)^[[0m

[Server:node1] ^[[31m    at org.infinispan.DecoratedCache.get(DecoratedCache.java:396)^[[0m

[Server:node1] ^[[31m    at org.infinispan.AbstractDelegatingCache.get(AbstractDelegatingCache.java:271)^[[0m

[Server:node1] ^[[31m    at org.jboss.as.clustering.infinispan.invoker.Locator$FindOperation.invoke(Locator.java:54)^[[0m

[Server:node1] ^[[31m    at org.jboss.as.clustering.infinispan.invoker.Locator$LockingFindOperation.invoke(Locator.java:71)^[[0m

[Server:node1] ^[[31m    at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)^[[0m

[Server:node1] ^[[31m    at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:82)^[[0m

[Server:node1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:109)^[[0m

[Server:node1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:55)^[[0m

[Server:node1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:149)^[[0m

[Server:node1] ^[[31m    at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:115)^[[0m

[Server:node1] ^[[31m    at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:677) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:node1] ^[[31m    at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:707) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:node1] ^[[31m    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:62) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:node1] ^[[31m    at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:node1] ^[[31m    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:node1] ^[[31m    at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)^[[0m

[Server:node1] ^[[31m    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:node1] ^[[31m    at org.wildfly.mod_cluster.undertow.metric.RunningRequestsHttpHandler.handleRequest(RunningRequestsHttpHandler.java:68)^[[0m

[Server:node1] ^[[31m    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:node1] ^[[31m    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:240) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:node1] ^[[31m    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:node1] ^[[31m    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:node1] ^[[31m    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:node1] ^[[31m    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177) [undertow-core-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:node1] ^[[31m    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727) [undertow-core-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:node1] ^[[31m    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]^[[0m

[Server:node1] ^[[31m    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]^[[0m

[Server:node1] ^[[31m    at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]^[[0m

[Server:node1] ^[[31m^[[0m

[Server:node1] ^[[31m^[[0m^[[31m07:50:52,290 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-14) ISPN000136: Execution error: java.lang.IllegalStateException: Transaction DummyTransaction{xid=DummyXid{id=6944}, status=1} is not in a valid state to be invoking cache operations on.^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.enlist(TxInterceptor.java:275)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.enlistIfNeeded(TxInterceptor.java:231)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:225)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:221)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:263)^[[0m

[Server:node1] ^[[31m    at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:247)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

3)

[Server:node1] ^[[0m^[[31m07:50:49,804 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (SessionExpirationScheduler - 1) ISPN000136: Execution error: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.lang.NullPointerException] while invoking method [public void org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.removed(org.infinispan.notifications.cachelistener.event.CacheEntryRemovedEvent)] on listener instance: org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager@325c50f7^[[0m

[Server:node1] ^[[31m    at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:211)^[[0m

[Server:node1] ^[[31m    at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22)^[[0m

[Server:node1] ^[[31m    at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation.invoke(AbstractListenerImpl.java:229)^[[0m

[Server:node1] ^[[31m    at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation.invoke(AbstractListenerImpl.java:192)^[[0m

[Server:node1] ^[[31m    at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryRemoved(CacheNotifierImpl.java:230)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.notify(RemoveCommand.java:96)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.performRemove(RemoveCommand.java:213)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.perform(RemoveCommand.java:92)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:91)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.distribution.TxDistributionInterceptor.handleTxWriteCommand(TxDistributionInterceptor.java:275)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRemoveCommand(TxDistributionInterceptor.java:80)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.CacheLoaderInterceptor.visitRemoveCommand(CacheLoaderInterceptor.java:132)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:321)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:402)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.EntryWrappingInterceptor.visitRemoveCommand(EntryWrappingInterceptor.java:216)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitRemoveCommand(PessimisticLockingInterceptor.java:145)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:255)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.visitRemoveCommand(TxInterceptor.java:196)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:206)^[[0m

[Server:node1] ^[[31m    at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:156)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:166)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:79)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)^[[0m

[Server:node1] ^[[31m    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)^[[0m

[Server:node1] ^[[31m    at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)^[[0m

[Server:node1] ^[[31m    at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1306)^[[0m

[Server:node1] ^[[31m    at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:407)^[[0m

[Server:node1] ^[[31m    at org.infinispan.CacheImpl.remove(CacheImpl.java:400)^[[0m

[Server:node1] ^[[31m    at org.infinispan.DecoratedCache.remove(DecoratedCache.java:406)^[[0m

[Server:node1] ^[[31m    at org.infinispan.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:281)^[[0m

[Server:node1] ^[[31m    at org.jboss.as.clustering.infinispan.invoker.Remover$RemoveOperation.invoke(Remover.java:49)^[[0m

[Server:node1] ^[[31m    at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)^[[0m

[Server:node1] ^[[31m    at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:82)^[[0m

[Server:node1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.remove(CoarseSessionFactory.java:117)^[[0m

[Server:node1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.remove(CoarseSessionFactory.java:55)^[[0m

[Server:node1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.InfinispanSession.invalidate(InfinispanSession.java:74)^[[0m

[Server:node1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.ExpiredSessionRemover.remove(ExpiredSessionRemover.java:47)^[[0m

[Server:node1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.ExpiredSessionRemover.remove(ExpiredSessionRemover.java:32)^[[0m

[Server:node1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.SessionExpirationScheduler$ExpirationTask.run(SessionExpirationScheduler.java:111)^[[0m

[Server:node1] ^[[31m    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51]^[[0m

[Server:node1] ^[[31m    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]^[[0m

[Server:node1] ^[[31m    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) [rt.jar:1.7.0_51]^[[0m

[Server:node1] ^[[31m    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) [rt.jar:1.7.0_51]^[[0m

[Server:node1] ^[[31m    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]^[[0m

[Server:node1] ^[[31m    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]^[[0m

[Server:node1] ^[[31m    at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]^[[0m

[Server:node1] ^[[31m    at org.jboss.threads.JBossThread.run(JBossThread.java:122)^[[0m

[Server:node1] ^[[31mCaused by: java.lang.NullPointerException^[[0m

[Server:node1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.coarse.CoarseImmutableSessionAttributes.getAttributeNames(CoarseImmutableSessionAttributes.java:51)^[[0m

[Server:node1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.removed(InfinispanSessionManager.java:239)^[[0m

[Server:node1] ^[[31m    at sun.reflect.GeneratedMethodAccessor1383.invoke(Unknown Source) [:1.7.0_51]^[[0m

[Server:node1] ^[[31m    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]^[[0m

[Server:node1] ^[[31m    at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]^[[0m

[Server:node1] ^[[31m    at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:207)^[[0m

[Server:node1] ^[[31m    ... 80 more^[[0m

[Server:node1] ^[[31m^[[0m

Can you please help us to understand these exceptions?

35. Re: Getting lots of exception related to infinispan while load testing

pferraro Aug 1, 2014 1:33 PM (in response to rishikesh_darandale)

The first exception is harmless. The session expiration thread acquires a lock on the session to ensure exclusive access. The lock acquisition failed because the session was being accessed.

The second exception is a bug. The RetryInvoker needs to be sensitive to the status of any current transaction. If the status is rollback-only, it cannot retry (and throws this exception).

[WFLY-3702] RetryInvoker is useless within the context of a transaction. - JBoss Issue Tracker

The last exception is fixed in the 8.x branch - and unfortunately was not merged in time for 8.1.0.Final.

https://github.com/wildfly/wildfly/commit/699fccbe495308b2f1f64409aa24829f89e0e138#diff-1f2397bce957971589c2b0cd241a1301

1 of 1 people found this helpful

36. Re: Getting lots of exception related to infinispan while load testing

rishikesh_darandale Aug 4, 2014 10:22 AM (in response to pferraro)

Thanks Paul,

How we should go about fixing last two exceptions?

Does using replicated cache is going to solve the issue?

37. Re: Getting lots of exception related to infinispan while load testing

rituraj Aug 4, 2014 1:08 PM (in response to rishikesh_darandale)

just want to make sure if we want to go with repl below will be new configuration ?

	<cache-container name="web" default-cache="repl" module="org.wildfly.clustering.web.infinispan">
	<transport cluster="${jboss.cluster.group.name}" lock-timeout="300000"/>
	<replicated-cache name="repl" batching="true" mode="SYNC" remote-timeout="300000">
	<state-transfer timeout="300000"/>
	<file-store/>
	</replicated-cache>
	</cache-container>

38. Re: Getting lots of exception related to infinispan while load testing

pferraro Aug 4, 2014 1:30 PM (in response to rishikesh_darandale)

To fix this you should cherry-pick the relevant commits and rebuild.

I just submitted a fix for WFLY-3702 here:

[8.x] WFLY-3702 RetryInvoker needs to be aware of transaction status by pferraro · Pull Request #6571 · wildfly/wildfly …

Using a replicated-cache vs a distributed-cache won't affect these exceptions. You would use one over the other based on your use case. If you have a small cluster, a replicated-cache is adequate and probably more efficient. If you have a large cluster (i.e. more than 4 nodes), a very dynamic cluster (i.e. the active number of nodes changes readily) or the size of the data to be stored in your cache is significant, a distributed-cache is typically preferable.

39. Re: Getting lots of exception related to infinispan while load testing

pferraro Aug 4, 2014 1:30 PM (in response to rituraj)

Sure - that's a valid replicated-cache configuration.

40. Re: Re: Getting lots of exception related to infinispan while load testing

rishikesh_darandale Sep 3, 2014 6:18 AM (in response to pferraro)

Hi Pual,

We have applied patch suggested by you on WF 8.1. But still we observed the exception related to infinispan dummy transaction.

Patch applied to WF 8.1

https://github.com/wildfly/wildfly/commit/a0d46de70d944627f3104c2ce103013e1e01fcf9

https://github.com/wildfly/wildfly/commit/3ca1b108d5208a1ee837a8c87d00f59ebe4ebe80

Merge pull request #6306 from pferraro/web-8.x · 699fccb · wildfly/wildfly · GitHub

Please find below stacktrace:

[Server:server1] ^[[31m^[[0m^[[31m01:29:28,464 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-41) ISPN000136: Execution error: java.lang.IllegalStateException: Transaction DummyTransaction{xid=DummyXid{id=364}, status=1} is not in a valid state to be invoking cache operations on.^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.enlist(TxInterceptor.java:275)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.enlistIfNeeded(TxInterceptor.java:231)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:225)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:221)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:server1] ^[[31m    at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:263)^[[0m

[Server:server1] ^[[31m    at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:247)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:92)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:79)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)^[[0m

[Server:server1] ^[[31m    at org.infinispan.CacheImpl.get(CacheImpl.java:377)^[[0m

[Server:server1] ^[[31m    at org.infinispan.DecoratedCache.get(DecoratedCache.java:396)^[[0m

[Server:server1] ^[[31m    at org.infinispan.AbstractDelegatingCache.get(AbstractDelegatingCache.java:271)^[[0m

[Server:server1] ^[[31m    at org.jboss.as.clustering.infinispan.invoker.Locator$FindOperation.invoke(Locator.java:54)^[[0m

[Server:server1] ^[[31m    at org.jboss.as.clustering.infinispan.invoker.Locator$LockingFindOperation.invoke(Locator.java:71)^[[0m

[Server:server1] ^[[31m    at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)^[[0m

[Server:server1] ^[[31m    at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:87)^[[0m

[Server:server1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:109)^[[0m

[Server:server1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:55)^[[0m

[Server:server1] ^[[31m    at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:149)^[[0m

[Server:server1] ^[[31m    at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:115)^[[0m

[Server:server1] ^[[31m    at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:677) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:server1] ^[[31m    at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:707) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:server1] ^[[31m    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:62) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:server1] ^[[31m    at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:server1] ^[[31m    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:server1] ^[[31m    at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)^[[0m

[Server:server1] ^[[31m    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:server1] ^[[31m    at org.wildfly.mod_cluster.undertow.metric.RunningRequestsHttpHandler.handleRequest(RunningRequestsHttpHandler.java:68)^[[0m

[Server:server1] ^[[31m    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:server1] ^[[31m    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:240) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:server1] ^[[31m    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:server1] ^[[31m    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:server1] ^[[31m    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:server1] ^[[31m    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177) [undertow-core-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:server1] ^[[31m    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727) [undertow-core-1.0.15.Final.jar:1.0.15.Final]^[[0m

[Server:server1] ^[[31m    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]^[[0m

[Server:server1] ^[[31m    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]^[[0m

[Server:server1] ^[[31m    at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]^[[0m

[Server:server1] ^[[31m^[[0m

[Server:server1] ^[[31m^[[0m^[[31m01:29:28,471 ERROR [io.undertow.request] (default task-41) UT005023: Exception handling request to /myapplication/part-number/A-1373-851-AMLP/0057/528: java.lang.IllegalStateException: Transaction DummyTransaction{xid=DummyXid{id=364}, status=1} is not in a valid state to be invoking cache operations on.^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.enlist(TxInterceptor.java:275)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.enlistIfNeeded(TxInterceptor.java:231)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:225)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:221)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:server1] ^[[31m    at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:263)^[[0m

[Server:server1] ^[[31m    at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:247)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:92)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:79)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)^[[0m

[Server:server1] ^[[31m    at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)^[[0m

[Server:server1] ^[[31m    at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)^[[0m

[Server:server1] ^[[31m    at org.infinispan.CacheImpl.get(CacheImpl.java:377)^[[0m

[Server:server1] ^[[31m    at org.infinispan.DecoratedCache.get(DecoratedCache.java:396)^[[0m

[Server:server1] ^[[31m    at org.infinispan.AbstractDelegatingCache.get(AbstractDelegatingCache.java:271)^[[0m

[Server:server1] ^[[31m    at org.jboss.as.clustering.infinispan.invoker.Locator$FindOperation.invoke(Locator.java:54)^[[0m

Are we missing anything?

Thanks for you help.

41. Re: Getting lots of exception related to infinispan while load testing

rituraj Sep 5, 2014 5:52 AM (in response to rishikesh_darandale)

Hi Paul,

we are seeing exceptions on 8.1.0 after applying the patch as well...and user are getting 500 error

we are able to reproduce this error but it is intermittently coming ...can you please help

[Server:server2] 04:16:48,572 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-21) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [15 seconds] on key [zJ2buOY0UrQ4sC9fIOsSsR1y] for requestor [GlobalTransaction:<pd_dc:server2/wildfly-consumer2-cluster>:1446:local]! Lock held by [GlobalTransaction:<pd_dc:server2/wildfly-consumer2-cluster>:1430:local]

[Server:server2] at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:198)

[Server:server2] at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:171)

[Server:server2] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:169)

[Server:server2] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitGetKeyValueCommand(PessimisticLockingInterceptor.java:70)

[Server:server2] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:server2] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:server2] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

[Server:server2] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

[Server:server2] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:server2] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:server2] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:226)

[Server:server2] at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:221)

[Server:server2] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:server2] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:server2] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

[Server:server2] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

[Server:server2] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:server2] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:server2] at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:263)

[Server:server2] at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:247)

[Server:server2] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

[Server:server2] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:server2] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:server2] at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:92)

[Server:server2] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:server2] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:server2] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)

[Server:server2] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)

[Server:server2] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

[Server:server2] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:server2] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:server2] at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:79)

[Server:server2] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

[Server:server2] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

[Server:server2] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)

[Server:server2] at org.infinispan.CacheImpl.get(CacheImpl.java:377)

[Server:server2] at org.infinispan.DecoratedCache.get(DecoratedCache.java:396)

[Server:server2] at org.infinispan.AbstractDelegatingCache.get(AbstractDelegatingCache.java:271)

[Server:server2] at org.jboss.as.clustering.infinispan.invoker.Locator$FindOperation.invoke(Locator.java:54)

[Server:server2] at org.jboss.as.clustering.infinispan.invoker.Locator$LockingFindOperation.invoke(Locator.java:71)

[Server:server2] at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)

[Server:server2] at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:87)

[Server:server2] at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:109)

[Server:server2] at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:55)

[Server:server2] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:149)

[Server:server2] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:115)

[Server:server2] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:677) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

[Server:server2] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:707) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

[Server:server2] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:62) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

[Server:server2] at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

[Server:server2] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

[Server:server2] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

[Server:server2] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

[Server:server2] at org.wildfly.mod_cluster.undertow.metric.RunningRequestsHttpHandler.handleRequest(RunningRequestsHttpHandler.java:68)

[Server:server2] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

[Server:server2] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:240) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

[Server:server2] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

[Server:server2] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

[Server:server2] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

[Server:server2] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

[Server:server2] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

[Server:server2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]

[Server:server2] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]

[Server:server2] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

[Server:server2]

[Server:server2] 04:16:48,598 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-21) ISPN000136: Execution error: java.lang.IllegalStateException: Transaction DummyTransaction{xid=DummyXid{id=1446}, status=1} is not in a valid state to be invoking cache operations on.

[Server:server2] at org.infinispan.interceptors.TxInterceptor.enlist(TxInterceptor.java:275)

[Server:server2] at org.infinispan.interceptors.TxInterceptor.enlistIfNeeded(TxInterceptor.java:231)

[Server:server2] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:225)