0 Replies Latest reply on Sep 9, 2014 6:48 AM by Alfie Kirkpatrick

    ConcurrentModificationException on cache get (from remote node)

    Alfie Kirkpatrick Newbie

      We're seeing the following exception occasionally on a production (loaded) cluster.

       

      I'm willing to believe this is an issue in our code, but would appreciate help understanding when/how this can happen given the stack trace below, and how to avoid it.

       

      This is a distributed cache with three nodes, single owner.

       

      My intepretation is that:

      - A cache entry is in the process of being serialized in response to request from remote node

      - At the same time a local modification is being made to the cache entry causing the CME

       

      If correct, this implies that changes to entries in the local cache are direct, ie. get + modification does not require a put for that modification to be seen by other local threads.

       

      What is the best way to resolve this issue? Does ISPN hold a synchronized lock on the cache entry during serialization, in which case I can wrap my modification in our own synchronized (entry) block?

       

      Any help/advice much appreciated.

       

      09-09-2014 07:36:52,823 ERROR [http-8080-68] com.evolok.ad.web.support.GeneralExceptionMapper  - An unknown error occurred processing request

      org.infinispan.commons.CacheException: java.util.ConcurrentModificationException

        at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:581)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:141)

        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:524)

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)

        at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.invokeClusterGetCommandRemotely(BaseDistributionInterceptor.java:129)

        at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.retrieveFromRemoteSource(BaseDistributionInterceptor.java:117)

        at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.remoteGetCacheEntry(NonTxDistributionInterceptor.java:161)

        at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitGetKeyValueCommand(NonTxDistributionInterceptor.java:48)

        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

        at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

        at org.infinispan.interceptors.CacheLoaderInterceptor.visitGetKeyValueCommand(CacheLoaderInterceptor.java:108)

        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

        at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:115)

        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

        at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitGetKeyValueCommand(NonTransactionalLockingInterceptor.java:34)

        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

        at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

        at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

        at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:258)

        at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:242)

        at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

        at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:92)

        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:106)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:70)

        at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)

        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:321)

        at org.infinispan.CacheImpl.get(CacheImpl.java:368)

        at org.infinispan.CacheImpl.get(CacheImpl.java:360)

        at com.evolok.ad.mixin.core.DefaultBuilder.load(DefaultBuilder.java:89)

        at com.evolok.ad.mixin.core.DefaultBuilder.load(DefaultBuilder.java:72)

        at com.evolok.ad.mixin.core.DefaultBuilder.load(DefaultBuilder.java:62)

        at com.evolok.ad.mixin.core.DefaultBuilder.activate(DefaultBuilder.java:45)

        at com.evolok.ad.session.SessionSource$SessionBuilderChainImpl.activate(SessionSource.java:205)

        at com.evolok.ad.session.spi.AbstractBuilder.activate(AbstractBuilder.java:15)

        at com.evolok.ad.session.SessionSource$SessionBuilderChainImpl.activate(SessionSource.java:205)

        at com.evolok.ad.session.SessionSource.activate(SessionSource.java:99)

        at com.evolok.ad.session.SessionSource.getMixin(SessionSource.java:56)

        at com.evolok.ad.meter.Meter.authorise(Meter.java:98)

        at com.evolok.ad.rules.AccessRule.evaluateCommands(AccessRule.java:259)

        at com.evolok.ad.rules.AccessRule.generateAccessRuleResult(AccessRule.java:171)

        at com.evolok.ad.rules.AccessRule.evaluate(AccessRule.java:113)

        at com.evolok.ad.rules.AccessRule.evaluateSubRules(AccessRule.java:209)

        at com.evolok.ad.rules.AccessRule.generateAccessRuleResult(AccessRule.java:147)

        at com.evolok.ad.rules.AccessRule.evaluate(AccessRule.java:113)

        at com.evolok.ad.rules.AccessRule.evaluateSubRules(AccessRule.java:209)

        at com.evolok.ad.rules.AccessRule.generateAccessRuleResult(AccessRule.java:147)

        at com.evolok.ad.rules.AccessRule.evaluate(AccessRule.java:113)

        at com.evolok.ad.rules.AccessRules.authorise(AccessRules.java:91)

        at com.evolok.ad.web.AuthorisationResource.authorize(AuthorisationResource.java:139)

        at com.evolok.ad.web.AuthorisationResource$AuthorizeCommand.process(AuthorisationResource.java:209)

        at com.evolok.ad.web.support.SessionRequestExecutor.with(SessionRequestExecutor.java:52)

        at com.evolok.ad.web.AuthorisationResource.authorizeEsi(AuthorisationResource.java:106)

        at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)

        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)

        at com.codahale.metrics.jersey.InstrumentedResourceMethodDispatchProvider$ExceptionMeteredRequestDispatcher.dispatch(InstrumentedResourceMethodDispatchProvider.java:69)

        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)

        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)

        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)

        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)

        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)

        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)

        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)

        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)

        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)

        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:98)

        at com.evolok.ad.web.security.ic.IdentityCoreSessionValidationFilter.doFilter(IdentityCoreSessionValidationFilter.java:85)

        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:102)

        at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:82)

        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)

        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)

        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:647)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)

        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

        at java.lang.Thread.run(Thread.java:662)

      Caused by: java.util.ConcurrentModificationException

        at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)

        at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:392)

        at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:391)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:678)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:680)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)

        at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)

        at org.infinispan.commons.marshall.MarshallUtil.marshallMap(MarshallUtil.java:36)

        at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:53)

        at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:27)

        at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:393)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)

        at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)

        at org.infinispan.container.entries.ImmortalCacheValue$Externalizer.writeObject(ImmortalCacheValue.java:125)

        at org.infinispan.container.entries.ImmortalCacheValue$Externalizer.writeObject(ImmortalCacheValue.java:122)

        at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:393)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)

        at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)

        at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.writeObject(SuccessfulResponse.java:71)

        at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.writeObject(SuccessfulResponse.java:64)

        at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:393)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)

        at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)

        at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:78)

        at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77)

        at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)

        at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)

        at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)

        at org.jgroups.blocks.RequestCorrelator.sendReply(RequestCorrelator.java:486)

        at org.jgroups.blocks.RequestCorrelator$ResponseImpl.send(RequestCorrelator.java:540)

        at org.infinispan.remoting.InboundInvocationHandlerImpl.reply(InboundInvocationHandlerImpl.java:220)

        at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:193)

        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:84)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:259)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:211)

        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)

        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)

        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)

        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)

        at org.jgroups.JChannel.up(JChannel.java:708)

        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)

        at org.jgroups.protocols.RSVP.up(RSVP.java:187)

        at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)

        at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)

        at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)

        at org.jgroups.protocols.tom.TOA.up(TOA.java:121)

        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)

        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)

        at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:694)

        at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:381)

        at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)

        at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)

        at org.jgroups.protocols.FD.up(FD.java:255)

        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)

        at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)

        at org.jgroups.protocols.Discovery.up(Discovery.java:379)

        at org.jgroups.protocols.TP.passMessageUp(TP.java:1399)

        at org.jgroups.protocols.TP$MyHandler.run(TP.java:1585)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        ... 1 more

      Caused by: an exception which occurred:

        in field viewed

        in field meters

        in field mixin

        in object com.evolok.ad.session.SessionHolder$MixinHolder@6b7950a6

        in field mixins

        in object com.evolok.ad.session.SessionHolder@1dca8e86

        in object org.infinispan.container.entries.ImmortalCacheValue@1dca8e86

        in object org.infinispan.remoting.responses.SuccessfulResponse@1dca8e86