Suspected member makes EJB call fail on other members

[ramiseesniblic]

Hello.

I am running JBoss EAP6 in a cluster of 3 nodes and I have noticed that when I fail one node (i.e. simulate a crash), it causes EJB exceptions on the other nodes which I believe is caused by Infinispan. (See stack trace below)

2016-03-14 16:11:50,837 ERROR [org.jboss.as.ejb3.invocation] (OCFWorkManager-threads - 43) JBAS014134: EJB Invocation failed on component CoreTransactionProcessor for method public abstract void xxx.business.services.ejb.interfaces.CoreTransactionProcessorLocal.processCa

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleInCallerTx(CMTTxInterceptor.java:162) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:252) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:341) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at xxx.business.services.ejb.interfaces.SessionServiceLocal$$$view61.notifySessionStateUpdate(Unknown Source) [RSEInstationServices.sar:]

        at xxx.business.services.handlers.data.extended.CameraSessionDataHandler.processCaptureRecord(CameraSessionDataHandler.java:104) [RSEInstationServices.sar:]

        at xxx.business.services.ejb.CoreTransactionProcessor.performActions(CoreTransactionProcessor.java:281) [RSEInstationServices.sar:]

        at xxx.business.services.ejb.CoreTransactionProcessor.processRecord(CoreTransactionProcessor.java:172) [RSEInstationServices.sar:]

        at xxx.business.services.ejb.CoreTransactionProcessor.processCaptureRecord(CoreTransactionProcessor.java:84) [RSEInstationServices.sar:]

        at sun.reflect.GeneratedMethodAccessor158.invoke(Unknown Source) [:1.7.0_65]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65]

        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65]

        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:250) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:341) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at xxx.business.services.ejb.interfaces.CoreTransactionProcessorLocal$$$view47.processCaptureRecord(Unknown Source) [RSEInstationServices.sar:]

        at xxx.business.iface.ocf.services.transformer.xxxx.OcfMessage.transformMessage(OcfSRMessage.java:204) [RSEInstationServices.sar:]

        at xxx.business.iface.ocf.services.transformer.common.OcfMessageProcessor.processOcfMessage(OcfMessageProcessor.java:72) [RSEInstationServices.sar:]

        at xxx.business.iface.ocf.services.mdb.xxxx.OCFSRListener.processMessage(OCFSRListener.java:56) [RSEInstationServices.sar:]

        at sun.reflect.GeneratedMethodAccessor162.invoke(Unknown Source) [:1.7.0_65]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65]

        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65]

        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:272) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:339) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:238) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponentDescription$5$1.processInvocation(MessageDrivenComponentDescription.java:203) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at xxx.ocf.jca.inbound.OCFListener$$$view10.processMessage(Unknown Source) [ocfrar.jar:]

        at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source) [:1.7.0_65]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65]

        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65]

        at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:140) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at com.sun.proxy.$Proxy46.processMessage(Unknown Source)

        at xxx.ocf.jca.spi.OCFXAResource.invokeMDB(OCFXAResource.java:107) [ocfrar.jar:]

        at xxx.ocf.jca.spi.OCFProcessor.invokeMDB(OCFProcessor.java:93) [ocfrar.jar:]

        at xxx.ocf.jca.spi.OCFProcessor.run(OCFProcessor.java:134) [ocfrar.jar:]

        at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:218) [ironjacamar-core-impl-1.0.26.Final-redhat-1.jar:1.0.26.Final-redhat-1]

        at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)

        at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:808)

        at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)

        at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:849)

        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65]

        at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Caused by: org.infinispan.remoting.transport.jgroups.SuspectException: Suspected member: rep1-node-02/jbosxxxa

        at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:88) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:541) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:173) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:194) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:251) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:238) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:233) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.releaseLocksOnFailureBeforePrepare(PessimisticLockingInterceptor.java:302) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPutKeyValueCommand(PessimisticLockingInterceptor.java:131) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:255) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:191) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:279) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:222) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:156) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:82) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1186) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.CacheImpl.putInternal(CacheImpl.java:783) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.CacheImpl.put(CacheImpl.java:777) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.DecoratedCache.put(DecoratedCache.java:320) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at org.infinispan.AbstractDelegatingCache.put(AbstractDelegatingCache.java:308) [infinispan-core-5.2.10.Final-redhat-1.jar:5.2.10.Final-redhat-1]

        at xxx.business.services.cache.common.Cache.putRecord(Cache.java:205) [RSEInstationServices.sar:]

        at xxx.business.services.handlers.session.CameraSessionHandler.updateRseStatusCache(CameraSessionHandler.java:995) [RSEInstationServices.sar:]

        at xxx.business.services.handlers.session.CameraSessionHandler.notifyNonRealtimeStateUpdate(CameraSessionHandler.java:796) [RSEInstationServices.sar:]

        at xxx.business.services.handlers.session.CameraSessionHandler.notifySessionStateUpdate(CameraSessionHandler.java:1350) [RSEInstationServices.sar:]

        at xxx.business.services.ejb.SessionService.notifySessionStateUpdate(SessionService.java:315) [RSEInstationServices.sar:]

        at sun.reflect.GeneratedMethodAccessor161.invoke(Unknown Source) [:1.7.0_65]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65]

        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65]

        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:250) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]

        ... 135 more

[wdfink]

Suspected members are still members of the cluster, so transactional access might fail.

Could you elaborate on the use-case?

- What kind of invocation

- what you do to fail a node

- could you share configuration (if not default) and code?

[ramiseesniblic]

Hello Wolf.

Thanks for you assistance.

Why would a transaction on another node in the cluster fail?  I understand that Infinispan and JGroups control the cluster but I would have thought that it would still operate if one node failed.

- What kind of invocation

     I am using a stateless session bean inside a transaction that has been invoked from a MDB.  Unusually, the MDB is invokes directly and not from a queue as I am using this method to link resource adaptor code to business logic code.  The stateless session bean uses other SSB, Infinispan caches and MySQL data sources to process and store date.

- What do you do to fail a node

     In this case, the Physical host that the VM is running on was switched off.  This was to simulate a crash scenario.

- Could you share configuration (if not default) and code?

     What configuration would you like?  Below is the JGroups configuration from the standalone.xml file:

        <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="tcp">

            <stack name="tcp">

                <transport type="TCP" socket-binding="jgroups-tcp">

  <property name="oob_thread_pool.enabled">true</property>

  <property name="oob_thread_pool.min_threads">1</property>

  <property name="oob_thread_pool.max_threads">300</property>

  </transport>

  <protocol type="TCPPING">

  <property name="initial_hosts">Node-01[7600],Node-02[7600],Node-03[7600]</property>

  <property name="timeout">3000</property>

  </protocol>

                <protocol type="MERGE2">

  <property name="min_interval">10000</property>

  <property name="max_interval">20000</property>

  </protocol>

                <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                <protocol type="FD"/>

                <protocol type="VERIFY_SUSPECT"/>

                <protocol type="pbcast.NAKACK"/>

                <protocol type="UNICAST2"/>

                <protocol type="pbcast.STABLE"/>

                <protocol type="pbcast.GMS"/>

            </stack>

        </subsystem>

    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="0">

        <socket-binding name="jgroups-tcp" port="7600"/>

        <socket-binding name="jgroups-tcp-fd" port="57600"/>

    </socket-binding-group>

[wdfink]

I assume that you use he cache direcly from your code, right?

So inside of a Tx it depends on the cache config whether other nodes are involved in your transaction.

So your Tx might span your node (SLSB) and one or more other where the cache entry is involved (dist or repl cache) so the cache is part of the transaction.

If now one node is suspected the Tx can't commit

[ramiseesniblic]

Hello Wolf.

Each of the nodes that run in the cluster are running the same code base and they all access the Infinispan cache directly using the org.infinispan.Cache<String, Object> Object.

Am I understanding correctly that because the cache that I think has the problem is configure as a clustered cache below, and is being accessed by all the nodes at

the same time, when one node fails the distributed transaction cannot be committed.

                <replicated-cache name="StatusCache" mode="SYNC">

                    <locking isolation="READ_COMMITTED" striping="false" acquire-timeout="20000"/>

                    <transaction stop-timeout="20000" mode="NON_DURABLE_XA" locking="PESSIMISTIC"/>

                </replicated-cache>

Q. Is there any configuration / change that can be made to prevent this scenario from occurring?

[wdfink]

If you don't need to share the cache between the node in your cluster you can use a <local> cache cofig.

If you need to share it you might use ASYNC or change the transaction mode