Why am I getting IllegalStateException thrown??

[drathnow]

I'm using Infinispan 7.1.1 with JBoss EAP 6.2. Things have been running along fine for a couple of weeks but then I started getting IllegalStateExceptions thrown and the exception looks like nothing I've seen before.   Could someone tell me what it means?

I hate including a full stack trace in a post but I'm not sure how much I can omit without making it useless so here is the whole thing:

02:27:27,617 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (10.70.130.78/17) ISPN000136: Execution error: java.lang.IllegalStateException: Transaction TransactionImple < ac, BasicAction: 0:ffffc0a80b03:444aa286:56588f2a:b110d6 status: ActionStatus.ABORTED > is not in a valid state to be invoking cache operations on.

  at org.infinispan.interceptors.TxInterceptor.enlist(TxInterceptor.java:362) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.interceptors.TxInterceptor.enlistIfNeeded(TxInterceptor.java:327) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:321) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:300) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:77) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:233) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:217) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:77) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.interceptors.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:103) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:91) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:77) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:423) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:415) [infinispan-core.jar:7.1.1.Final]

  at org.infinispan.query.impl.EntityLoader.load(EntityLoader.java:28) [infinispan-query.jar:7.1.1.Final]

  at org.infinispan.query.impl.EntityLoader.load(EntityLoader.java:34) [infinispan-query.jar:7.1.1.Final]

  at org.infinispan.query.impl.CacheQueryImpl.list(CacheQueryImpl.java:162) [infinispan-query.jar:7.1.1.Final]

  at zedi.pacbridge.app.cache.InfinispanOutgoingRequestCache.outgoingRequestsForSiteAddress(InfinispanOutgoingRequestCache.java:84) [pacbridge-app-6.0.0.jar:]

  at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) [:1.7.0_79]

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [rt.jar:1.7.0_79]

  at java.lang.reflect.Method.invoke(Unknown Source) [rt.jar:1.7.0_79]

  at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:129) [jboss-as-weld-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:137) [jboss-as-weld-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47) [jboss-as-jpa-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:94) [jboss-as-weld-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:272) [jboss-as-ejb3-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:339) [jboss-as-ejb3-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [jboss-as-ejb3-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

  at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]

  at zedi.pacbridge.app.controls.OutgoingRequestCache$$$view3.outgoingRequestsForSiteAddress(Unknown Source) [pacbridge-app-6.0.0.jar:]

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_79]

  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) [rt.jar:1.7.0_79]

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [rt.jar:1.7.0_79]

  at java.lang.reflect.Method.invoke(Unknown Source) [rt.jar:1.7.0_79]

  at org.jboss.weld.util.reflection.SecureReflections$13.work(SecureReflections.java:267) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]

  at org.jboss.weld.util.reflection.SecureReflectionAccess.run(SecureReflectionAccess.java:52) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]

  at org.jboss.weld.util.reflection.SecureReflectionAccess.runAsInvocation(SecureReflectionAccess.java:137) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]

  at org.jboss.weld.util.reflection.SecureReflections.invoke(SecureReflections.java:263) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]

  at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:115) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]

  at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]

  at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105) [weld-core-1.1.16.Final-redhat-1.jar:1.1.16.Final-redhat-1]

  at zedi.pacbridge.app.cache.OutgoingRequestCache$OutgoingRequestCacheUpdateDelegate$-2135361606$Proxy$_$$_Weld$Proxy$.outgoingRequestsForSiteAddress(OutgoingRequestCache$OutgoingRequestCacheUpdateDelegate$-2135361606$Proxy$_$$_Weld$Proxy$.java) [pacbridge-app-6.0.0.jar:]

  at zedi.pacbridge.app.net.ConnectionManagerHelper$WorkerRunner.run(ConnectionManagerHelper.java:72) [pacbridge-app-6.0.0.jar:]

  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [rt.jar:1.7.0_79]

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.7.0_79]

  at java.lang.Thread.run(Unknown Source) [rt.jar:1.7.0_79]

[rvansa]

Each transaction that's open for more than (by default) 1 minute is aborted by transaction manager. When you call commit/rollback on such aborted transaction, this error is thrown (since it's not in a state when you could call even rollback).

Another question is why it was hanging for so long - operations in Infinispan are limited by certain timeout (depends on your configuration), but if you e.g. use pessimistic transactions and execute many contended writes during the transaction, it could take a while.

[drathnow]

That might explain something else that happened before this.  I have a number of worker threads in my app that handle communications with remote devices.  Depending on how much work there is, the number of worker threads will grow up to a an upper limit. About 2-3 min before I started seeing this exception, my app started adding worker threads until it hit the limit.  Unfortunately, I wasn't notified of the problem until after JBoss was restarted, which cleared the problem.  If it happens again, I'll try to get a thread dump.

Thanks,