6 Replies Latest reply on May 25, 2018 6:08 AM by galder.zamarreno

    Infinispan concurrent modification exception after restarting a node

    carlosra85

      Hi,

       

      I just restart a second node in my cluster configuration, and I start getting errors for the infinispan:

       

      12:12:57,249 DEBUG [org.jboss.as.config] (MSC service thread 1-3) Configured system properties:

      JTAEnvironmentBean.xaAssumeRecoveryComplete = true

      [Standalone] =

      awt.toolkit = sun.awt.X11.XToolkit

      file.encoding = UTF-8

      file.encoding.pkg = sun.io

      file.separator = /

      java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment

      java.awt.headless = true

      java.awt.printerjob = sun.print.PSPrinterJob

      java.class.path = /opt/java/ModulabWildFly/wildfly/jboss-modules.jar

      java.class.version = 52.0

      java.endorsed.dirs = /usr/java/jdk1.8.0_151/jre/lib/endorsed

      java.ext.dirs = /usr/java/jdk1.8.0_151/jre/lib/ext:/usr/java/packages/lib/ext

      java.home = /usr/java/jdk1.8.0_151/jre

      java.io.tmpdir = /tmp

      java.library.path = /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

      java.net.preferIPv4Stack = true

      java.runtime.name = Java(TM) SE Runtime Environment

      java.runtime.version = 1.8.0_151-b12

      java.specification.name = Java Platform API Specification

      java.specification.vendor = Oracle Corporation

      java.specification.version = 1.8

      java.util.logging.manager = org.jboss.logmanager.LogManager

      java.vendor = Oracle Corporation

      java.vendor.url = http://java.oracle.com/

      java.vendor.url.bug = http://bugreport.sun.com/bugreport/

      java.version = 1.8.0_151

      java.vm.info = mixed mode

      java.vm.name = Java HotSpot(TM) 64-Bit Server VM

      java.vm.specification.name = Java Virtual Machine Specification

      java.vm.specification.vendor = Oracle Corporation

      java.vm.specification.version = 1.8

      java.vm.vendor = Oracle Corporation

      java.vm.version = 25.151-b12

      javax.management.builder.initial = org.jboss.as.jmx.PluggableMBeanServerBuilder

      javax.xml.datatype.DatatypeFactory = __redirected.__DatatypeFactory

      javax.xml.parsers.DocumentBuilderFactory = __redirected.__DocumentBuilderFactory

      javax.xml.parsers.SAXParserFactory = __redirected.__SAXParserFactory

      javax.xml.stream.XMLEventFactory = __redirected.__XMLEventFactory

      javax.xml.stream.XMLInputFactory = __redirected.__XMLInputFactory

      javax.xml.stream.XMLOutputFactory = __redirected.__XMLOutputFactory

      javax.xml.transform.TransformerFactory = __redirected.__TransformerFactory

      javax.xml.validation.SchemaFactory:http://www.w3.org/2001/XMLSchema = __redirected.__SchemaFactory

      javax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom = __redirected.__XPathFactory

      jboss.home.dir = /opt/java/ModulabWildFly/wildfly

      jboss.host.name = mov-jeapmgold-302

      jboss.modules.dir = /opt/java/ModulabWildFly/wildfly/modules

      jboss.modules.system.pkgs = org.jboss.byteman

      jboss.node.name = node1

      jboss.qualified.host.name = mov-jeapmgold-302.salud.psis

      jboss.server.base.dir = /opt/java/ModulabWildFly/wildfly/standalone

      jboss.server.config.dir = /opt/java/ModulabWildFly/wildfly/standalone/configuration

      jboss.server.data.dir = /opt/java/ModulabWildFly/wildfly/standalone/data

      jboss.server.deploy.dir = /opt/java/ModulabWildFly/wildfly/standalone/data/content

      jboss.server.log.dir = /opt/java/ModulabWildFly/wildfly/standalone/log

      jboss.server.name = mov-jeapmgold-302

      jboss.server.persist.config = true

      jboss.server.temp.dir = /opt/java/ModulabWildFly/wildfly/standalone/tmp

      line.separator =

       

       

      logging.configuration = file:/opt/java/ModulabWildFly/wildfly/standalone/configuration/logging.properties

      module.path = /opt/java/ModulabWildFly/wildfly/modules

      org.jboss.boot.log.file = /opt/java/ModulabWildFly/wildfly/standalone/log/server.log

      org.jboss.resolver.warning = true

      org.xml.sax.driver = __redirected.__XMLReaderFactory

      os.arch = amd64

      os.name = Linux

      os.version = 3.10.0-693.5.2.el7.x86_64

      path.separator = :

      startModulabScheduler =

      sun.arch.data.model = 64

      sun.boot.class.path = /usr/java/jdk1.8.0_151/jre/lib/resources.jar:/usr/java/jdk1.8.0_151/jre/lib/rt.jar:/usr/java/jdk1.8.0_151/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_151/jre/lib/jsse.jar:/usr/java/jdk1.8.0_151/jre/lib/jce.jar:/usr/java/jdk1.8.0_151/jre/lib/charsets.jar:/usr/java/jdk1.8.0_151/jre/lib/jfr.jar:/usr/java/jdk1.8.0_151/jre/classes

      sun.boot.library.path = /usr/java/jdk1.8.0_151/jre/lib/amd64

      sun.cpu.endian = little

      sun.cpu.isalist =

      sun.io.unicode.encoding = UnicodeLittle

      sun.java.command = /opt/java/ModulabWildFly/wildfly/jboss-modules.jar -mp /opt/java/ModulabWildFly/wildfly/modules org.jboss.as.standalone -Djboss.home.dir=/opt/java/ModulabWildFly/wildfly -Djboss.server.base.dir=/opt/java/ModulabWildFly/wildfly/standalone -c standalone-full-ha.xml

      sun.java.launcher = SUN_STANDARD

      sun.jnu.encoding = UTF-8

      sun.management.compiler = HotSpot 64-Bit Tiered Compilers

      sun.os.patch.level = unknown

      user.country = US

      user.dir = /opt/java/ModulabWildFly/wildfly

      user.home = /export/home/werfen

      user.language = en

      user.name = werfen

      user.timezone = Europe/Madrid

      12:12:57,266 DEBUG [org.jboss.as.config] (MSC service thread 1-3) VM Arguments: -D[Standalone] -Xms512M -Xmx2048M -XX:MaxMetaspaceSize=512M -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -DJTAEnvironmentBean.xaAssumeRecoveryComplete=true -Djboss.node.name=node1 -DstartModulabScheduler -Dorg.jboss.boot.log.file=/opt/java/ModulabWildFly/wildfly/standalone/log/server.log -Dlogging.configuration=file:/opt/java/ModulabWildFly/wildfly/standalone/configuration/logging.properties

      12:12:58,929 WARN  [org.jboss.as.server.deployment.scanner] (ServerService Thread Pool -- 6) WFLYDS0006: Reliable deployment behaviour is not possible when auto-deployment of exploded content is enabled (i.e. deployment without use of ".dodeploy"' marker files). Configuration of auto-deployment of exploded content is not recommended in any situation where reliability is desired. Configuring the deployment scanner's auto-deploy-exploded setting to "false" is recommended.

      12:13:03,003 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-4) JGRP000015: the send buffer of socket ManagedMulticastSocketBinding was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

      12:13:03,013 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-4) JGRP000015: the receive buffer of socket ManagedMulticastSocketBinding was set to 20MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

      12:13:03,048 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-4) JGRP000015: the send buffer of socket ManagedMulticastSocketBinding was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

      12:13:03,049 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-4) JGRP000015: the receive buffer of socket ManagedMulticastSocketBinding was set to 25MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

      12:13:10,415 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-1) WFLYSRV0018: Deployment "deployment.modulab.ear.ModulabGoldServerWeb.war" is using a private module ("org.jboss.as.jmx:main") which may be changed or removed in future versions without notice.

      12:13:14,357 WARN  [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (MSC service thread 1-2) IJ000407: No lazy enlistment available for jdbc/OracleDS

      12:13:28,349 WARNING [org.jgroups.protocols.UDP] (default task-31) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

      12:13:28,354 WARNING [org.jgroups.protocols.UDP] (default task-31) JGRP000015: the receive buffer of socket MulticastSocket was set to 20MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

      12:13:28,355 WARNING [org.jgroups.protocols.UDP] (default task-31) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

      12:13:28,361 WARNING [org.jgroups.protocols.UDP] (default task-31) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

      12:13:37,773 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (default task-9) ISPN000073: Unexpected error while replicating: org.infinispan.commons.CacheException: java.util.ConcurrentModificationException

      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:158)

      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:138)

      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:611)

      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:144)

      at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:185)

      at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:197)

      at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:187)

      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)

      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:37)

      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:93)

      at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:83)

      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:158)

      at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:145)

      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

      at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

      at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)

      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)

      at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)

      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

      at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114)

      at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:83)

      at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:67)

      at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

      at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)

      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)

      at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)

      at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:121)

      at org.infinispan.transaction.impl.TransactionCoordinator.prepare(TransactionCoordinator.java:104)

      at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:46)

      at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)

      at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368)

      at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91)

      at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)

      at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)

      at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)

      at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)

      at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1722)

      at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1679)

      at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1121)

      at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1111)

      at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1742)

      at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:248)

      at com.systelab.modulabgold.server.common.access.StationCache.updateConnectionStatusBySessionID(StationCache.java:144)

      at com.systelab.modulabgold.server.connection.NewRTMessageXML.runAction(NewRTMessageXML.java:141)

      at com.systelab.modulabgold.server.common.xml.XMLBase.executeActions(XMLBase.java:508)

      at com.systelab.modulabgold.server.common.xml.XMLBase.process(XMLBase.java:377)

      at com.systelab.modulabgold.server.connection.NewRTMessageXML.process(NewRTMessageXML.java:93)

      at com.systelab.modulabgold.server.connection.NewRTMessageServlet.service(NewRTMessageServlet.java:18)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

      at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)

      at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:201)

      at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:178)

      at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

      at com.systelab.modulabgold.server.filter.CORSFilter.doFilter(CORSFilter.java:34)

      at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

      at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

      at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

      at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

      at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

      at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

      at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

      at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

      at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

      at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

      at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

      at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

      at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)

      at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)

      at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)

      at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)

      at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)

      at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)

      at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)

      at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

      at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)

      at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

      at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

      Caused by: java.util.ConcurrentModificationException

      at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:907)

      at java.util.ArrayList$Itr.next(ArrayList.java:857)

      at org.infinispan.commons.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:173)

      at org.infinispan.marshall.exts.ListExternalizer.writeObject(ListExternalizer.java:42)

      at org.infinispan.marshall.exts.ListExternalizer.writeObject(ListExternalizer.java:24)

      at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:485)

      at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

      at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)

      at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)

      at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)

      at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

      at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

      at org.infinispan.commons.marshall.MarshallUtil.marshallMap(MarshallUtil.java:44)

      at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:60)

      at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:28)

      at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:485)

      at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

      at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

      at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

      at org.infinispan.marshall.DeltaAwareObjectOutput.writeObject(DeltaAwareObjectOutput.java:29)

      at org.infinispan.commands.write.PutKeyValueCommand.writeTo(PutKeyValueCommand.java:120)

      at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:67)

      at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:53)

      at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:41)

      at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:485)

      at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

      at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

      at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

      at org.infinispan.marshall.DeltaAwareObjectOutput.writeObject(DeltaAwareObjectOutput.java:29)

      at org.infinispan.commons.marshall.MarshallUtil.marshallArray(MarshallUtil.java:126)

      at org.infinispan.commands.tx.PrepareCommand.writeTo(PrepareCommand.java:197)

      at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:67)

      at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:138)

      at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:125)

      at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:68)

      at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:485)

      at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

      at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

      at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

      at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:72)

      at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77)

      at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)

      at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)

      at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)

      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:275)

      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:393)

      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:133)

      ... 116 more

      Caused by: an exception which occurred:

      in field multionlineConnections

      in object com.systelab.modulabgold.common.data.StationData@1d246665

      in object java.util.HashMap@706d9e90

      in object org.infinispan.commands.write.PutKeyValueCommand@5594c871

      in object org.infinispan.commands.tx.PrepareCommand@a7718894

       

       

      I attach my infinispan config and the jgroups config:

       

      <?xml version="1.0" encoding="UTF-8"?>

      <infinispan

              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

              xsi:schemaLocation="urn:infinispan:config:8.2 http://www.infinispan.org/schemas/infinispan-config-8.2.xsd"

              xmlns="urn:infinispan:config:8.2">

        <jgroups>

            <stack-file name="external-file" path="com/systelab/modulabgold/server/common/infinispan/infinispan-jgroups.xml"/>

        </jgroups>

               <cache-container name="modulab_cache_container" default-cache="default">

                 <transport cluster="${jboss.partition.name:defaultInfinispanCluster}" stack="external-file" lock-timeout="60000"/>

                 <replicated-cache name="license_cache" mode="SYNC">

         <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

                     <transaction locking="OPTIMISTIC" mode="BATCH"/>

                     <eviction strategy="NONE"/>

                 </replicated-cache>

                 <replicated-cache name="logout_sessions_cache" mode="SYNC">

         <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

                     <transaction locking="OPTIMISTIC" mode="BATCH"/>

                     <eviction strategy="NONE"/>

                 </replicated-cache>

                 <replicated-cache name="stations_cache" mode="SYNC">

         <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

                     <transaction locking="OPTIMISTIC" mode="BATCH"/>

                     <eviction strategy="NONE"/>

                 </replicated-cache>

                 <replicated-cache name="jobs_cache" mode="SYNC">

         <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

                     <transaction locking="OPTIMISTIC" mode="BATCH"/>

                     <eviction strategy="NONE"/>

                 </replicated-cache>

                 <replicated-cache name="modulab_globals_cache" mode="SYNC">

         <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

                     <transaction locking="OPTIMISTIC" mode="BATCH"/>

                     <eviction strategy="NONE"/>

                 </replicated-cache>

                 <replicated-cache name="vector_alerts_cache" mode="SYNC">

         <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

                     <transaction locking="OPTIMISTIC" mode="BATCH"/>

                     <eviction strategy="NONE"/>

                 </replicated-cache>

                 <distributed-cache name="modulab_cache" mode="SYNC">

         <locking isolation="REPEATABLE_READ" striping="false" acquire-timeout="60000"/>

                     <transaction locking="OPTIMISTIC" mode="NON_XA"/>

                     <expiration interval="600000" lifespan="600000" max-idle="600000"/>

                 </distributed-cache>

             </cache-container>

      </infinispan>

       

      Thanks for any help!

        • 1. Re: Infinispan concurrent modification exception after restarting a node
          pferraro

          You seem to be using native Infinispan and JGroups configuration using jars bundled with your application, rather than the Infinispan/JGroups versions/configuration used by WildFly.  Consequently, you should direct this question to the appropriate forum:   Infinispan

          • 2. Re: Infinispan concurrent modification exception after restarting a node
            galder.zamarreno

            Agree with Paul. Seems like this collection might be referenced by multiple entities stored in Infinispan? Avoid the sharing and it'd be fine.

            • 3. Re: Infinispan concurrent modification exception after restarting a node
              carlosra85

              After these type of errors I start getting these ones and the server logs are quickly filled and the server stops responding. Can this be related to the multiple references and the possible sharing?

               

              2018-05-17 14:30:11,628 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (Transaction Reaper Worker 446) ISPN000136: Error executing command RollbackCommand, writing keys [stations]: org.infinispan.commons.CacheException: Thread interrupted while invoking RPC

              at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)

              at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitRollbackCommand(TxDistributionInterceptor.java:208)

              at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

              at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)

              at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

              at org.infinispan.interceptors.NotificationInterceptor.visitRollbackCommand(NotificationInterceptor.java:51)

              at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

              at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitRollbackCommand(AbstractTxLockingInterceptor.java:53)

              at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

              at org.infinispan.interceptors.TxInterceptor.visitRollbackCommand(TxInterceptor.java:197)

              at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

              at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)

              at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitRollbackCommand(TransactionSynchronizerInterceptor.java:66)

              at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

              at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)

              at org.infinispan.statetransfer.StateTransferInterceptor.visitRollbackCommand(StateTransferInterceptor.java:97)

              at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

              at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)

              at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

              at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:114)

              at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:83)

              at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)

              at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

              at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:67)

              at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)

              at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

              at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:117)

              at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:40)

              at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)

              at org.infinispan.transaction.impl.TransactionCoordinator.rollbackInternal(TransactionCoordinator.java:231)

              at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:170)

              at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:67)

              at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)

              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:542)

              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:473)

              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:127)

              at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:215)

              at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:381)

              at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78)

              Caused by: java.lang.InterruptedException

              at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347)

              at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)

              at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:100)

              at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:187)

              ... 52 more

               

              2018-05-17 14:30:11,378 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-70) ISPN000136: Error executing command RollbackCommand, writing keys [stations]: org.infinispan.util.concurrent.TimeoutException: Replication timeout for hrcrpant0158-53056

              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:801)

              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:642)

              at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)

              at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)

              at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

              at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)

              at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)

              at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)

              at java.util.concurrent.FutureTask.run(FutureTask.java:266)

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

              at java.lang.Thread.run(Thread.java:745)

               

               

              Thanks a lot!

              • 4. Re: Infinispan concurrent modification exception after restarting a node
                wdfink

                Did you checked whether GC is an issue? Often the long GC pauses cause several issues.

                • 5. Re: Infinispan concurrent modification exception after restarting a node
                  carlosra85

                  Can this be a result of a bad configuration of the jgroups? With UPD the replication errors are frequent. With TCP and the initial_hosts of the cluster seems that the error does not happen.

                  • 6. Re: Infinispan concurrent modification exception after restarting a node
                    galder.zamarreno

                    Hmmm, that sounds like maybe UDP communications are slow in your network, maybe as result of misconfiguration? JGroups comes with some performance tests you can run on your network. You should run those standalone tests in your environment and get in touch with the network experts.