1 2 Previous Next 16 Replies Latest reply on Jul 4, 2018 1:31 AM by Prakash Kolandaivelu

    Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

    Prakash Kolandaivelu Newbie

      Upgraded the infinispan from 6.0.2.Final to 9.2.4.Final. All seems to be working when the cluster is started. But after recovering from split brain, below exception is thrown and cluster stops working. No code change done after the upgrade in the clustering.

       

      Infinispan 6 config.

      -------------------

          <infinispan>

          <global>

              <transport>

              <properties>

              <property name="configurationFile" value="configuration/jgroups.xml"/>

              </properties>

          </transport>

          <!-- Enable JMX statistics -->

          <globalJmxStatistics

              enabled="true"

              jmxDomain="org.infinispan"

              cacheManagerName="CacheManager"/>

          </global>

          <default>

          <!-- Configure a synchronous replication cache -->

          <clustering mode="replication">

            <sync/>

          </clustering>

          <!--

              Used to register JMX statistics in any available MBean server

          -->

          <jmxStatistics enabled="true"/>

          </default>

          <!-- transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" -->

          <namedCache name="transactional-type">

          <transaction

              transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup"

              syncRollbackPhase="true"

              syncCommitPhase="true"

              cacheStopTimeout="30000"

              use1PcForAutoCommitTransactions="true"

              autoCommit="true"

              lockingMode="OPTIMISTIC"

              useSynchronization="true"

              transactionMode="TRANSACTIONAL"

              />

          </namedCache>

          </infinispan>

       

      Infinispan 9 config.

      -----------------------

          <?xml version="1.0" encoding="UTF-8"?>

          <infinispan>

          <jgroups>

              <stack-file name="tcp" path="configuration/jgroups.xml"/>

          </jgroups>

          <cache-container default-cache="default">

          <transport stack="tcp" cluster="cluster" channel="other"/>

          <replicated-cache name="default" statistics="true"  mode="ASYNC" remote-timeout="20000" >

              <locking

                  isolation="READ_COMMITTED" striping="false" acquire-timeout="180000"

                  write-skew="true"/>

              <versioning scheme="SIMPLE"/>

          </replicated-cache>

          <replicated-cache name="transactional-type" statistics="true"  mode="SYNC" remote-timeout="20000" >

              <locking

                   isolation="READ_COMMITTED" striping="false" acquire-timeout="180000"

                   write-skew="true"/>

              <transaction

                   locking="OPTIMISTIC"

                   auto-commit="true"

                   complete-timeout="60000"

                   mode="FULL_XA"

                   notifications="true"

                   stop-timeout="30000"

                   transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup"/>

          </replicated-cache>

          </cache-container>

          </infinispan>

       

      jgroup.xml after the upgrade.

      -----------------------------

          <config xmlns="urn:org:jgroups"

              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

              xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-4.0.xsd">

          <TCP bind_addr="${jgroups.tcp.address}"

               bind_port="12000" port_range="10"

               recv_buf_size="20000000"

               send_buf_size="640000"

               bundler_type="no-bundler"

               enable_diagnostics="true"

               thread_naming_pattern="cl"

               thread_pool.min_threads="0"

               thread_pool.max_threads="8"

               thread_pool.keep_alive_time="5000"

          />

          <TCPGOSSIP initial_hosts="${jgroups.tcpgossip.initial_hosts}"/>

          <MERGE3/>

          <FD timeout="3000" max_tries="3"/>

          <VERIFY_SUSPECT timeout="500"/>

          <pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true" />

          <UNICAST3/>

          <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000" max_bytes="10m"/>

          <pbcast.GMS print_local_addr="true" join_timeout="${jgroups.join_timeout:5000}"

                      max_bundling_time="30"

                      view_bundling="true"/>

          <MFC max_credits="2M" min_threshold="0.4"/>

          <FRAG3/>

          </config>

       

      Everything seems to be working including the merge after the split brain recovery, Below exception occurs after recovering from split brain. this does not happens consistently. 1 out of 4 times this happens.

       

          2018-06-12 16:15:21.748 IST [jgroups-302,ndb-cluster,ndbDevBox-39617] ERROR org.infinispan.interceptors.impl.InvocationContextInterceptor - ISPN000136: Error executing command PrepareCommand, writing keys [admin]

          org.infinispan.remoting.RemoteException: ISPN000217: Received exception from DevBox-43633, see cause for remote stack trace

          at org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

          at org.infinispan.remoting.transport.impl.MapResponseCollector.addException(MapResponseCollector.java:65) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

          at org.infinispan.remoting.transport.impl.MapResponseCollector$IgnoreLeavers.addException(MapResponseCollector.java:103) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

          at org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

          at org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

          at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52) ~[na:na]

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

          at org.jgroups.JChannel.up(JChannel.java:819) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.protocols.FRAG3.up(FRAG3.java:171) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:343) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.protocols.pbcast.GMS.up(GMS.java:864) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.protocols.FD.up(FD.java:212) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.protocols.MERGE3.up(MERGE3.java:276) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.protocols.Discovery.up(Discovery.java:267) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.protocols.TP.passMessageUp(TP.java:1248) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87) [jgroups-4.0.11.Final.jar:4.0.11.Final]

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]

          at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

          Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 10 seconds for key admin and requestor RecoveryAwareGlobalTransaction{xid=Xid{formatId=131077, globalTransactionId=00000000000000000000FFFF7F00010100008DB35B1F966000000D0E31,branchQualifier=00000000000000000000FFFF7F00010100008DB35B1F966000000D0F0000000000000000}, internalId=1688854155231238} GlobalTx:DevBox-39617:837. Lock is held by RecoveryAwareGlobalTransaction{xid=Xid{formatId=131077, globalTransactionId=00000000000000000000FFFF7F0001010000BA065B1F967200000FCE31,branchQualifier=00000000000000000000FFFF7F0001010000BA065B1F967200000FCF0000000000000000}, internalId=1688854155231236} GlobalTx:DevBox-43633:1012

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:288) ~[na:na]

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:218) ~[na:na]

          at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.checkState(InfinispanLock.java:436) ~[na:na]

          at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.toInvocationStage(InfinispanLock.java:408) ~[na:na]

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.toInvocationStage(DefaultLockManager.java:248) ~[na:na]

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.toInvocationStage(DefaultLockManager.java:272) ~[na:na]

          at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:53) ~[na:na]

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185) ~[na:na]

          at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:183) ~[na:na]

          at org.infinispan.interceptors.impl.TxInterceptor.handlePrepareCommand(TxInterceptor.java:137) ~[na:na]

          at org.infinispan.interceptors.impl.TxInterceptor.visitPrepareCommand(TxInterceptor.java:126) ~[na:na]

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185) ~[na:na]

          at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

          at org.infinispan.interceptors.BaseAsyncInterceptor.lambda$new$0(BaseAsyncInterceptor.java:22) ~[na:na]

          at org.infinispan.interceptors.InvocationSuccessFunction.apply(InvocationSuccessFunction.java:21) ~[na:na]

          at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:69) ~[na:na]

          at org.infinispan.interceptors.InvocationStage.thenApply(InvocationStage.java:24) ~[na:na]

          at org.infinispan.interceptors.BaseAsyncInterceptor.asyncInvokeNext(BaseAsyncInterceptor.java:224) ~[na:na]

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:46) ~[na:na]

          at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) ~[na:na]

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:203) ~[na:na]

          at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:69) ~[na:na]

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185) ~[na:na]

          at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

          at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) ~[na:na]

          at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:132) ~[na:na]

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185) ~[na:na]

          at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) ~[na:na]

          at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) ~[na:na]

          at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]

          at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) ~[na:na]

          at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:132) ~[na:na]

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185) ~[na:na]

          at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) ~[na:na]

          at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:234) ~[na:na]

          at org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:109) ~[na:na]

          at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94) ~[na:na]

          at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99) ~[na:na]

          at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71) ~[na:na]

          at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40) ~[na:na]

          ... 3 common frames omitted

       

      Cluster members can change the same key.

      Need help in fixing this exception. Not sure if this was discussed any thread. could not find any thing so posted it.

       

      Any code change is required in the way cache update ?

      Is this the configuration mistake or anything else need to be corrected?

        • 1. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery
          Dan Berindei Expert

          I'm not 100% sure but I think it's a combination of having a rather small JGroups thread pool and Infinispan 9.2.4 performing conflict resolution on merge by default. BTW, we've reverted that in 9.3.0.Beta1, you now have to request conflict resolution explicitly -- we should probably backport ISPN-9103 to 9.2.x.

           

          I suggest disabling conflict resolution with <partition-handling merge-policy="NONE"/> first. If you still have problems, increase thread_pool.max_threads (e.g. 32). If you still have problems then, please collect some thread dumps and collect TRACE (or at least DEBUG) logs from org.infinispan so we can investigate further.

          • 2. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery
            Prakash Kolandaivelu Newbie

            Hi Dan,

             

            Thanks for talking time and replying.

             

            We have migrated from infinispan 6.0.2.Final to 9.2.4.Final to solve the split brain merge. Our product is based on the ODL hydrogen code base based. Turning off the merge is not possible as it has lot of data structure to be synced. calling resolveConflicts() on all the data structure will be tidious task.

             

            Below are options we tried out from your reply.

            increased thread_pool.max_threads to 32 and increased the lock timeout to 180 secs as the back end operation might take 180 sec for performing the operation. With this the above issue is suppressed for now. Will be keep testing to check this occurs again.

             

            We are hitting the below issue when try to access the empty map. any sugesstions ?

            2018-06-13 22:16:14.436 PDT [http-nio-8443-exec-9] ERROR org.infinispan.interceptors.impl.InvocationContextInterceptor - ISPN000136: Error executing command SizeCommand, writing keys []

            java.lang.NullPointerException: null

                at org.infinispan.commons.hash.MurmurHash3.hash(MurmurHash3.java:399) ~[infinispan-commons-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.distribution.ch.impl.HashFunctionPartitioner.getSegment(HashFunctionPartitioner.java:45) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.stream.impl.AbstractCacheStream.isPrimaryOwner(AbstractCacheStream.java:425) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.stream.impl.tx.TxDistributedCacheStream.lambda$null$0(TxDistributedCacheStream.java:68) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) ~[na:1.8.0_131]

                at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1620) ~[na:1.8.0_131]

                at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_131]

                at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_131]

                at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_131]

                at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_131]

                at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_131]

                at org.infinispan.stream.impl.tx.TxDistributedCacheStream.lambda$supplierForSegments$2(TxDistributedCacheStream.java:70) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.stream.impl.termop.SegmentRetryingOperation.performOperation(SegmentRetryingOperation.java:78) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.stream.impl.AbstractCacheStream.performOperationRehashAware(AbstractCacheStream.java:288) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.stream.impl.AbstractCacheStream.performOperation(AbstractCacheStream.java:232) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.stream.impl.DistributedCacheStream.count(DistributedCacheStream.java:426) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:45) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:20) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.impl.CallInterceptor.visitCommand(CallInterceptor.java:29) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]

                at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.visitSizeCommand(BaseDistributionInterceptor.java:115) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

                at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

                at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

                at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

                at org.infinispan.interceptors.impl.TxInterceptor.visitSizeCommand(TxInterceptor.java:270) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

                at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]

                at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:347) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

                at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) ~[na:na]

                at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) ~[na:na]

                at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]

                at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:449) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:444) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                at java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:1082) [na:1.8.0_131]

                at java.util.concurrent.ConcurrentHashMap.<init>(ConcurrentHashMap.java:852) [na:1.8.0_131]

                at org.opendaylight.controller.sal.implementation.internal.FlowProgrammerService.removeFlows(FlowProgrammerService.java:511) [org.opendaylight.controller.sal.implementation-0.5.10-SNAPSHOT.jar:na]

                at com.comp.prod.monitor.internal.Monitor.toggleRuleInstall(Monitor.java:5809) [com.comp.prod.monitor-3.1.9-SNAPSHOT.jar:na]

                at com.comp.one.apps.monitor.web.MonitorWeb.toggleRule(MonitorWeb.java:3200) [com.comp.one.apps.monitor.web-0.6.10-SNAPSHOT.jar:na]

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]

                at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]

                at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219) [org.springframework.org.springframework.web-3.1.3.RELEASE.jar:3.1.3.RELEASE]

                at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) [org.springframework.org.springframework.web-3.1.3.RELEASE.jar:3.1.3.RELEASE]

                at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:100) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]

                at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:604) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]

                at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:565) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]

                at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]

                at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]

                at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]

                at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]

                at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]

                at javax.servlet.http.HttpServlet.service(HttpServlet.java:755) [equinoxSDK381.javax.servlet-3.0.0.v201112011016.jar:na]

                at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) [equinoxSDK381.javax.servlet-3.0.0.v201112011016.jar:na]

                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]

                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]

                at com.comp.one.apps.monitor.web.cors.CorsFilter.doFilter(CorsFilter.java:65) [com.comp.one.apps.monitor.web-0.6.10-SNAPSHOT.jar:na]

                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]

                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]

                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]

                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]

                at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]

                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]

                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]

                at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:341) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]

                at com.comp.prod.urlredirect.URLRedirectValve.invoke(URLRedirectValve.java:35) [com.comp.prod.urlredirectvalve-0.5.11-SNAPSHOT.jar:na]

                at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]

                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]

                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]

                at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041) [orbit.org.apache.coyote-7.0.47.v201312060745.jar:na]

                at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603) [orbit.org.apache.coyote-7.0.47.v201312060745.jar:na]

                at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1721) [orbit.org.apache.coyote-7.0.47.v201312060745.jar:na]

                at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679) [orbit.org.apache.coyote-7.0.47.v201312060745.jar:na]

                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]

                at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

             

            Thanks,Prakash

            • 3. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery
              Dan Berindei Expert

              The stack trace implies that there is an entry with a null key in the transaction context (cache.size() counts both entries in the cache and entries modified in the current transaction but not yet committed). AFAIK this should never happen, can you tell us what other cache methods your application calls in the same transaction as cache.size()?

              • 5. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery
                Prakash Kolandaivelu Newbie

                size() is the only operation performed on the empty map. To narrow down.. i stopped all other operation and verified it.  Thanks for confirming the bug. That really saves my time.

                 

                Is there any safe bed, with partition handing, where we will not hit this issue ?

                 

                I am not sure this is related to the same issue. Below execption also happened on the empty map. The operation performed is .values() on the empty map.

                2018-06-17 13:00:22.356 PDT [Redirection Interfaces Monitor] ERROR org.infinispan.interceptors.impl.InvocationContextInterceptor - ISPN000136: Error executing command SizeCommand, writing keys []

                org.infinispan.util.concurrent.TimeoutException: null

                    at org.infinispan.stream.impl.AbstractCacheStream.performOperationRehashAware(AbstractCacheStream.java:313) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.stream.impl.AbstractCacheStream.performOperation(AbstractCacheStream.java:232) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.stream.impl.DistributedCacheStream.count(DistributedCacheStream.java:426) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:45) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:20) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.impl.CallInterceptor.visitCommand(CallInterceptor.java:29) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]

                    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.visitSizeCommand(BaseDistributionInterceptor.java:115) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

                    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

                    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

                    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

                    at org.infinispan.interceptors.impl.TxInterceptor.visitSizeCommand(TxInterceptor.java:270) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

                    at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]

                    at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:347) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

                    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) ~[na:na]

                    at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) ~[na:na]

                    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]

                    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:449) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:444) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.commands.read.AbstractCloseableIteratorCollection.size(AbstractCloseableIteratorCollection.java:47) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at org.infinispan.util.CollectionMapper.size(CollectionMapper.java:34) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

                    at java.util.HashSet.<init>(HashSet.java:118) [na:1.8.0_131]

                    at com.comp.prod.monitor.internal.Monitor.redirectionInterfacesMonitor(Monitor.java:8873) [com.comp.prod.monitor-3.1.9-SNAPSHOT.jar:na]

                    at com.comp.prod.monitor.internal.Monitor$2.run(Monitor.java:8638) [com.comp.prod.monitor-3.1.9-SNAPSHOT.jar:na]

                    at java.util.TimerThread.mainLoop(Timer.java:555) [na:1.8.0_131]

                    at java.util.TimerThread.run(Timer.java:505) [na:1.8.0_131]

                • 6. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery
                  Dan Berindei Expert

                  As a workaround for the original NullPointerException, you can change the isolation level to REPEATABLE_READ.

                   

                  I'm not sure about the TimeoutException, it's very generic so I'd need some information: a thread dump and/or trace logs for org.infinispan.

                  • 7. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery
                    Prakash Kolandaivelu Newbie

                    Upgraded to 9.3.0. I get the below error.

                    [ERROR] Bundle org.opendaylight.controller:clustering.services-implementation:bundle:0.5.11-SNAPSHOT : Classes found in the wrong directory: {META-INF/versions/10/org/infinispan/commons/jdkspecific/ProcessorInfo.class=org.infinispan.commons.jdkspecific.ProcessorInfo, META-INF/versions/10/org/infinispan/commons/jdkspecific/CallerId.class=org.infinispan.commons.jdkspecific.CallerId}

                     

                    is there a way to come out of it.

                    • 8. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery
                      Tristan Tarrant Master

                      You need to upgrade your maven plugins to newer versions that support

                      multi-release jars

                      • 9. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery
                        Prakash Kolandaivelu Newbie

                        REPEATABLE_READ worked. Since the upgrade is not successful i am using this.

                        • 11. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery
                          Prakash Kolandaivelu Newbie

                          Yes upgrading it to maven-bundle-plugin 3.4.0 with 9.3.0 Final worked.. thanks.. let me test it and update

                          • 12. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery
                            Prakash Kolandaivelu Newbie

                            Looks like we have to upgrade the osgi. we are using 3.8. Is there a way without upgrading the osgi we make it work ?

                            • 13. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery
                              Tristan Tarrant Master

                              You will need to filter out the META-INF/versions/* path from the

                              infinispan-commons.jar

                              But I really recommend getting your environment ready for jdk 11, as many

                              more components will use multi-release jars to ensure backwards/forward

                              compatibility.

                              • 14. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery
                                Prakash Kolandaivelu Newbie

                                What worked for me?

                                Upgraded to infinispan 9.3.0 Final with the below changes.

                                 

                                We had  org.eclipse.osgi-3.8.1.v20120830, this does not support multi-release jars. Updated the maven-bundle-plugin and added <_noee>true</_noee> under configuration so avoid the Require-Capability tag in the Manifest file.

                                osgi - In bndtools, what causes a Require-Capability header for execution environment to be generated? - Stack Overflow

                                 

                                Thanks for the Infinispan team support.

                                1 2 Previous Next