Infinispan concurrent modification exception after restarting a node

[carlosra85]

Hi,

I just restart a second node in my cluster configuration, and I start getting errors for the infinispan:

12:12:57,249 DEBUG [org.jboss.as.config] (MSC service thread 1-3) Configured system properties:

JTAEnvironmentBean.xaAssumeRecoveryComplete = true

[Standalone] =

awt.toolkit = sun.awt.X11.XToolkit

file.encoding = UTF-8

file.encoding.pkg = sun.io

file.separator = /

java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment

java.awt.headless = true

java.awt.printerjob = sun.print.PSPrinterJob

java.class.path = /opt/java/ModulabWildFly/wildfly/jboss-modules.jar

java.class.version = 52.0

java.endorsed.dirs = /usr/java/jdk1.8.0_151/jre/lib/endorsed

java.ext.dirs = /usr/java/jdk1.8.0_151/jre/lib/ext:/usr/java/packages/lib/ext

java.home = /usr/java/jdk1.8.0_151/jre

java.io.tmpdir = /tmp

java.library.path = /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

java.net.preferIPv4Stack = true

java.runtime.name = Java(TM) SE Runtime Environment

java.runtime.version = 1.8.0_151-b12

java.specification.name = Java Platform API Specification

java.specification.vendor = Oracle Corporation

java.specification.version = 1.8

java.util.logging.manager = org.jboss.logmanager.LogManager

java.vendor = Oracle Corporation

java.vendor.url = http://java.oracle.com/

java.vendor.url.bug = http://bugreport.sun.com/bugreport/

java.version = 1.8.0_151

java.vm.info = mixed mode

java.vm.name = Java HotSpot(TM) 64-Bit Server VM

java.vm.specification.name = Java Virtual Machine Specification

java.vm.specification.vendor = Oracle Corporation

java.vm.specification.version = 1.8

java.vm.vendor = Oracle Corporation

java.vm.version = 25.151-b12

javax.management.builder.initial = org.jboss.as.jmx.PluggableMBeanServerBuilder

javax.xml.datatype.DatatypeFactory = __redirected.__DatatypeFactory

javax.xml.parsers.DocumentBuilderFactory = __redirected.__DocumentBuilderFactory

javax.xml.parsers.SAXParserFactory = __redirected.__SAXParserFactory

javax.xml.stream.XMLEventFactory = __redirected.__XMLEventFactory

javax.xml.stream.XMLInputFactory = __redirected.__XMLInputFactory

javax.xml.stream.XMLOutputFactory = __redirected.__XMLOutputFactory

javax.xml.transform.TransformerFactory = __redirected.__TransformerFactory

javax.xml.validation.SchemaFactory:http://www.w3.org/2001/XMLSchema = __redirected.__SchemaFactory

javax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom = __redirected.__XPathFactory

jboss.home.dir = /opt/java/ModulabWildFly/wildfly

jboss.host.name = mov-jeapmgold-302

jboss.modules.dir = /opt/java/ModulabWildFly/wildfly/modules

jboss.modules.system.pkgs = org.jboss.byteman

jboss.node.name = node1

jboss.qualified.host.name = mov-jeapmgold-302.salud.psis

jboss.server.base.dir = /opt/java/ModulabWildFly/wildfly/standalone

jboss.server.config.dir = /opt/java/ModulabWildFly/wildfly/standalone/configuration

jboss.server.data.dir = /opt/java/ModulabWildFly/wildfly/standalone/data

jboss.server.deploy.dir = /opt/java/ModulabWildFly/wildfly/standalone/data/content

jboss.server.log.dir = /opt/java/ModulabWildFly/wildfly/standalone/log

jboss.server.name = mov-jeapmgold-302

jboss.server.persist.config = true

jboss.server.temp.dir = /opt/java/ModulabWildFly/wildfly/standalone/tmp

line.separator =

logging.configuration = file:/opt/java/ModulabWildFly/wildfly/standalone/configuration/logging.properties

module.path = /opt/java/ModulabWildFly/wildfly/modules

org.jboss.boot.log.file = /opt/java/ModulabWildFly/wildfly/standalone/log/server.log

org.jboss.resolver.warning = true

org.xml.sax.driver = __redirected.__XMLReaderFactory

os.arch = amd64

os.name = Linux

os.version = 3.10.0-693.5.2.el7.x86_64

path.separator = :

startModulabScheduler =

sun.arch.data.model = 64

sun.boot.class.path = /usr/java/jdk1.8.0_151/jre/lib/resources.jar:/usr/java/jdk1.8.0_151/jre/lib/rt.jar:/usr/java/jdk1.8.0_151/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_151/jre/lib/jsse.jar:/usr/java/jdk1.8.0_151/jre/lib/jce.jar:/usr/java/jdk1.8.0_151/jre/lib/charsets.jar:/usr/java/jdk1.8.0_151/jre/lib/jfr.jar:/usr/java/jdk1.8.0_151/jre/classes

sun.boot.library.path = /usr/java/jdk1.8.0_151/jre/lib/amd64

sun.cpu.endian = little

sun.cpu.isalist =

sun.io.unicode.encoding = UnicodeLittle

sun.java.command = /opt/java/ModulabWildFly/wildfly/jboss-modules.jar -mp /opt/java/ModulabWildFly/wildfly/modules org.jboss.as.standalone -Djboss.home.dir=/opt/java/ModulabWildFly/wildfly -Djboss.server.base.dir=/opt/java/ModulabWildFly/wildfly/standalone -c standalone-full-ha.xml

sun.java.launcher = SUN_STANDARD

sun.jnu.encoding = UTF-8

sun.management.compiler = HotSpot 64-Bit Tiered Compilers

sun.os.patch.level = unknown

user.country = US

user.dir = /opt/java/ModulabWildFly/wildfly

user.home = /export/home/werfen

user.language = en

user.name = werfen

user.timezone = Europe/Madrid

12:12:57,266 DEBUG [org.jboss.as.config] (MSC service thread 1-3) VM Arguments: -D[Standalone] -Xms512M -Xmx2048M -XX:MaxMetaspaceSize=512M -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -DJTAEnvironmentBean.xaAssumeRecoveryComplete=true -Djboss.node.name=node1 -DstartModulabScheduler -Dorg.jboss.boot.log.file=/opt/java/ModulabWildFly/wildfly/standalone/log/server.log -Dlogging.configuration=file:/opt/java/ModulabWildFly/wildfly/standalone/configuration/logging.properties

12:12:58,929 WARN  [org.jboss.as.server.deployment.scanner] (ServerService Thread Pool -- 6) WFLYDS0006: Reliable deployment behaviour is not possible when auto-deployment of exploded content is enabled (i.e. deployment without use of ".dodeploy"' marker files). Configuration of auto-deployment of exploded content is not recommended in any situation where reliability is desired. Configuring the deployment scanner's auto-deploy-exploded setting to "false" is recommended.

12:13:03,003 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-4) JGRP000015: the send buffer of socket ManagedMulticastSocketBinding was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

12:13:03,013 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-4) JGRP000015: the receive buffer of socket ManagedMulticastSocketBinding was set to 20MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

12:13:03,048 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-4) JGRP000015: the send buffer of socket ManagedMulticastSocketBinding was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

12:13:03,049 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-4) JGRP000015: the receive buffer of socket ManagedMulticastSocketBinding was set to 25MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

12:13:10,415 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-1) WFLYSRV0018: Deployment "deployment.modulab.ear.ModulabGoldServerWeb.war" is using a private module ("org.jboss.as.jmx:main") which may be changed or removed in future versions without notice.

12:13:14,357 WARN  [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (MSC service thread 1-2) IJ000407: No lazy enlistment available for jdbc/OracleDS

12:13:28,349 WARNING [org.jgroups.protocols.UDP] (default task-31) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

12:13:28,354 WARNING [org.jgroups.protocols.UDP] (default task-31) JGRP000015: the receive buffer of socket MulticastSocket was set to 20MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

12:13:28,355 WARNING [org.jgroups.protocols.UDP] (default task-31) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

12:13:28,361 WARNING [org.jgroups.protocols.UDP] (default task-31) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

12:13:37,773 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (default task-9) ISPN000073: Unexpected error while replicating: org.infinispan.commons.CacheException: java.util.ConcurrentModificationException

at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:158)

at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:138)

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:611)

at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:144)

at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:185)

at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:197)

at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:187)

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:37)

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:93)

at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:83)

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:158)

at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:145)

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)

at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114)

at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:83)

at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:67)

at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)

at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:121)

at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:104)

at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:46)

at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)

at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)

at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)

at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)

at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)

at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)

at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)

at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1722)

at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1679)

at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1121)

at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1111)

at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1742)

at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:248)

at com.systelab.modulabgold.server.common.access.StationCache.updateConnectionStatusBySessionID(StationCache.java:144)

at com.systelab.modulabgold.server.connection.NewRTMessageXML.runAction(NewRTMessageXML.java:141)

at com.systelab.modulabgold.server.common.xml.XMLBase.executeActions(XMLBase.java:508)

at com.systelab.modulabgold.server.common.xml.XMLBase.process(XMLBase.java:377)

at com.systelab.modulabgold.server.connection.NewRTMessageXML.process(NewRTMessageXML.java:93)

at com.systelab.modulabgold.server.connection.NewRTMessageServlet.service(NewRTMessageServlet.java:18)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:201)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:178)

at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

at com.systelab.modulabgold.server.filter.CORSFilter.doFilter(CORSFilter.java:34)

at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)

at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)

at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)

at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)

at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)

at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)

at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)

at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)

at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.util.ConcurrentModificationException

at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:907)

at java.util.ArrayList$Itr.next(ArrayList.java:857)

at org.infinispan.commons.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:173)

at org.infinispan.marshall.exts.ListExternalizer.writeObject(ListExternalizer.java:42)

at org.infinispan.marshall.exts.ListExternalizer.writeObject(ListExternalizer.java:24)

at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:485)

at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)

at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)

at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)

at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

at org.infinispan.commons.marshall.MarshallUtil.marshallMap(MarshallUtil.java:44)

at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:60)

at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:28)

at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:485)

at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

at org.infinispan.marshall.DeltaAwareObjectOutput.writeObject(DeltaAwareObjectOutput.java:29)

at org.infinispan.commands.write.PutKeyValueCommand.writeTo(PutKeyValueCommand.java:120)

at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:67)

at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:53)

at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:41)

at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:485)

at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

at org.infinispan.marshall.DeltaAwareObjectOutput.writeObject(DeltaAwareObjectOutput.java:29)

at org.infinispan.commons.marshall.MarshallUtil.marshallArray(MarshallUtil.java:126)

at org.infinispan.commands.tx.PrepareCommand.writeTo(PrepareCommand.java:197)

at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:67)

at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:138)

at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:125)

at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:68)

at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:485)

at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:72)

at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77)

at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)

at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)

at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)

at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:275)

at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:393)

at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:133)

... 116 more

Caused by: an exception which occurred:

in field multionlineConnections

in object com.systelab.modulabgold.common.data.StationData@1d246665

in object java.util.HashMap@706d9e90

in object org.infinispan.commands.write.PutKeyValueCommand@5594c871

in object org.infinispan.commands.tx.PrepareCommand@a7718894

I attach my infinispan config and the jgroups config:

<?xml version="1.0" encoding="UTF-8"?>

<infinispan

        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

        xsi:schemaLocation="urn:infinispan:config:8.2 http://www.infinispan.org/schemas/infinispan-config-8.2.xsd"

        xmlns="urn:infinispan:config:8.2">

  <jgroups>

      <stack-file name="external-file" path="com/systelab/modulabgold/server/common/infinispan/infinispan-jgroups.xml"/>

  </jgroups>

         <cache-container name="modulab_cache_container" default-cache="default">

           <transport cluster="${jboss.partition.name:defaultInfinispanCluster}" stack="external-file" lock-timeout="60000"/>

           <replicated-cache name="license_cache" mode="SYNC">

   <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

               <transaction locking="OPTIMISTIC" mode="BATCH"/>

               <eviction strategy="NONE"/>

           </replicated-cache>

           <replicated-cache name="logout_sessions_cache" mode="SYNC">

   <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

               <transaction locking="OPTIMISTIC" mode="BATCH"/>

               <eviction strategy="NONE"/>

           </replicated-cache>

           <replicated-cache name="stations_cache" mode="SYNC">

   <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

               <transaction locking="OPTIMISTIC" mode="BATCH"/>

               <eviction strategy="NONE"/>

           </replicated-cache>

           <replicated-cache name="jobs_cache" mode="SYNC">

   <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

               <transaction locking="OPTIMISTIC" mode="BATCH"/>

               <eviction strategy="NONE"/>

           </replicated-cache>

           <replicated-cache name="modulab_globals_cache" mode="SYNC">

   <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

               <transaction locking="OPTIMISTIC" mode="BATCH"/>

               <eviction strategy="NONE"/>

           </replicated-cache>

           <replicated-cache name="vector_alerts_cache" mode="SYNC">

   <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

               <transaction locking="OPTIMISTIC" mode="BATCH"/>

               <eviction strategy="NONE"/>

           </replicated-cache>

           <distributed-cache name="modulab_cache" mode="SYNC">

   <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

               <transaction locking="OPTIMISTIC" mode="NON_XA"/>

               <expiration interval="600000" lifespan="600000" max-idle="600000"/>

           </distributed-cache>

       </cache-container>

</infinispan>

Thanks for any help!

[pferraro]

You seem to be using native Infinispan and JGroups configuration using jars bundled with your application, rather than the Infinispan/JGroups versions/configuration used by WildFly.  Consequently, you should direct this question to the appropriate forum:   Infinispan

[galder.zamarreno]

Agree with Paul. Seems like this collection might be referenced by multiple entities stored in Infinispan? Avoid the sharing and it'd be fine.

[carlosra85]

After these type of errors I start getting these ones and the server logs are quickly filled and the server stops responding. Can this be related to the multiple references and the possible sharing?

2018-05-17 14:30:11,628 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (Transaction Reaper Worker 446) ISPN000136: Error executing command RollbackCommand, writing keys [stations]: org.infinispan.commons.CacheException: Thread interrupted while invoking RPC

at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)

at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:208)

at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)

at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:51)

at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:53)

at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:197)

at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)

at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:66)

at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)

at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:97)

at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)

at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114)

at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:83)

at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)

at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:67)

at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)

at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)

at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)

at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:231)

at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:170)

at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:67)

at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)

at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:542)

at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:473)

at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:127)

at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:215)

at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:381)

at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78)

Caused by: java.lang.InterruptedException

at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347)

at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)

at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:100)

at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:187)

... 52 more

2018-05-17 14:30:11,378 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-70) ISPN000136: Error executing command RollbackCommand, writing keys [stations]: org.infinispan.util.concurrent.TimeoutException: Replication timeout for hrcrpant0158-53056

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)

at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)

at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)

at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)

at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Thanks a lot!

[wdfink]

Did you checked whether GC is an issue? Often the long GC pauses cause several issues.

[carlosra85]

Can this be a result of a bad configuration of the jgroups? With UPD the replication errors are frequent. With TCP and the initial_hosts of the cluster seems that the error does not happen.

[galder.zamarreno]

Hmmm, that sounds like maybe UDP communications are slow in your network, maybe as result of misconfiguration? JGroups comes with some performance tests you can run on your network. You should run those standalone tests in your environment and get in touch with the network experts.