1 2 Previous Next 16 Replies Latest reply on Jul 4, 2018 1:31 AM by pkolanda

Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

pkolanda Jun 12, 2018 11:39 AM

Upgraded the infinispan from 6.0.2.Final to 9.2.4.Final. All seems to be working when the cluster is started. But after recovering from split brain, below exception is thrown and cluster stops working. No code change done after the upgrade in the clustering.

Infinispan 6 config.

-------------------

</properties>

</transport>

<globalJmxStatistics

enabled="true"

jmxDomain="org.infinispan"

cacheManagerName="CacheManager"/>

</global>

<sync/>

</clustering>

<!--

Used to register JMX statistics in any available MBean server

-->

</default>

<transaction

transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup"

syncRollbackPhase="true"

syncCommitPhase="true"

cacheStopTimeout="30000"

use1PcForAutoCommitTransactions="true"

autoCommit="true"

lockingMode="OPTIMISTIC"

useSynchronization="true"

transactionMode="TRANSACTIONAL"

</namedCache>

</infinispan>

Infinispan 9 config.

-----------------------

<?xml version="1.0" encoding="UTF-8"?>

<stack-file name="tcp" path="configuration/jgroups.xml"/>

</jgroups>

<cache-container default-cache="default">

<replicated-cache name="default" statistics="true" mode="ASYNC" remote-timeout="20000" >

<locking

isolation="READ_COMMITTED" striping="false" acquire-timeout="180000"

write-skew="true"/>

</replicated-cache>

<replicated-cache name="transactional-type" statistics="true" mode="SYNC" remote-timeout="20000" >

<locking

isolation="READ_COMMITTED" striping="false" acquire-timeout="180000"

write-skew="true"/>

<transaction

locking="OPTIMISTIC"

auto-commit="true"

complete-timeout="60000"

mode="FULL_XA"

notifications="true"

stop-timeout="30000"

transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup"/>

</replicated-cache>

</cache-container>

</infinispan>

jgroup.xml after the upgrade.

-----------------------------

<config xmlns="urn:org:jgroups"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-4.0.xsd">

<TCP bind_addr="${jgroups.tcp.address}"

bind_port="12000" port_range="10"

recv_buf_size="20000000"

send_buf_size="640000"

bundler_type="no-bundler"

enable_diagnostics="true"

thread_naming_pattern="cl"

thread_pool.min_threads="0"

thread_pool.max_threads="8"

thread_pool.keep_alive_time="5000"

<VERIFY_SUSPECT timeout="500"/>

<pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true" />

<pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000" max_bytes="10m"/>

<pbcast.GMS print_local_addr="true" join_timeout="${jgroups.join_timeout:5000}"

max_bundling_time="30"

view_bundling="true"/>

</config>

Everything seems to be working including the merge after the split brain recovery, Below exception occurs after recovering from split brain. this does not happens consistently. 1 out of 4 times this happens.

2018-06-12 16:15:21.748 IST [jgroups-302,ndb-cluster,ndbDevBox-39617] ERROR org.infinispan.interceptors.impl.InvocationContextInterceptor - ISPN000136: Error executing command PrepareCommand, writing keys [admin]

org.infinispan.remoting.RemoteException: ISPN000217: Received exception from DevBox-43633, see cause for remote stack trace

at org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

at org.infinispan.remoting.transport.impl.MapResponseCollector.addException(MapResponseCollector.java:65) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

at org.infinispan.remoting.transport.impl.MapResponseCollector$IgnoreLeavers.addException(MapResponseCollector.java:103) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

at org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

at org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]

at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52) ~[na:na]

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1318) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1221) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1356) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]

at org.jgroups.JChannel.up(JChannel.java:819) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.protocols.FRAG3.up(FRAG3.java:171) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.protocols.FlowControl.up(FlowControl.java:343) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.protocols.pbcast.GMS.up(GMS.java:864) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.protocols.FD.up(FD.java:212) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.protocols.MERGE3.up(MERGE3.java:276) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.protocols.Discovery.up(Discovery.java:267) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.protocols.TP.passMessageUp(TP.java:1248) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87) [jgroups-4.0.11.Final.jar:4.0.11.Final]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]

at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 10 seconds for key admin and requestor RecoveryAwareGlobalTransaction{xid=Xid{formatId=131077, globalTransactionId=00000000000000000000FFFF7F00010100008DB35B1F966000000D0E31,branchQualifier=00000000000000000000FFFF7F00010100008DB35B1F966000000D0F0000000000000000}, internalId=1688854155231238} GlobalTx:DevBox-39617:837. Lock is held by RecoveryAwareGlobalTransaction{xid=Xid{formatId=131077, globalTransactionId=00000000000000000000FFFF7F0001010000BA065B1F967200000FCE31,branchQualifier=00000000000000000000FFFF7F0001010000BA065B1F967200000FCF0000000000000000}, internalId=1688854155231236} GlobalTx:DevBox-43633:1012

at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:288) ~[na:na]

at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:218) ~[na:na]

at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.checkState(InfinispanLock.java:436) ~[na:na]

at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.toInvocationStage(InfinispanLock.java:408) ~[na:na]

at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.toInvocationStage(DefaultLockManager.java:248) ~[na:na]

at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.toInvocationStage(DefaultLockManager.java:272) ~[na:na]

at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:53) ~[na:na]

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185) ~[na:na]

at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:183) ~[na:na]

at org.infinispan.interceptors.impl.TxInterceptor.handlePrepareCommand(TxInterceptor.java:137) ~[na:na]

at org.infinispan.interceptors.impl.TxInterceptor.visitPrepareCommand(TxInterceptor.java:126) ~[na:na]

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185) ~[na:na]

at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

at org.infinispan.interceptors.BaseAsyncInterceptor.lambda$new$0(BaseAsyncInterceptor.java:22) ~[na:na]

at org.infinispan.interceptors.InvocationSuccessFunction.apply(InvocationSuccessFunction.java:21) ~[na:na]

at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:69) ~[na:na]

at org.infinispan.interceptors.InvocationStage.thenApply(InvocationStage.java:24) ~[na:na]

at org.infinispan.interceptors.BaseAsyncInterceptor.asyncInvokeNext(BaseAsyncInterceptor.java:224) ~[na:na]

at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:46) ~[na:na]

at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185) ~[na:na]

at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:203) ~[na:na]

at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:69) ~[na:na]

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185) ~[na:na]

at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]

at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) ~[na:na]

at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:132) ~[na:na]

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185) ~[na:na]

at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) ~[na:na]

at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) ~[na:na]

at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]

at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) ~[na:na]

at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:132) ~[na:na]

at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185) ~[na:na]

at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) ~[na:na]

at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:234) ~[na:na]

at org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:109) ~[na:na]

at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94) ~[na:na]

at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99) ~[na:na]

at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71) ~[na:na]

at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40) ~[na:na]

... 3 common frames omitted

Cluster members can change the same key.

Need help in fixing this exception. Not sure if this was discussed any thread. could not find any thing so posted it.

Any code change is required in the way cache update ?

Is this the configuration mistake or anything else need to be corrected?

1. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

dan.berindei Jun 13, 2018 3:16 AM (in response to pkolanda)

I'm not 100% sure but I think it's a combination of having a rather small JGroups thread pool and Infinispan 9.2.4 performing conflict resolution on merge by default. BTW, we've reverted that in 9.3.0.Beta1, you now have to request conflict resolution explicitly -- we should probably backport ISPN-9103 to 9.2.x.

I suggest disabling conflict resolution with <partition-handling merge-policy="NONE"/> first. If you still have problems, increase thread_pool.max_threads (e.g. 32). If you still have problems then, please collect some thread dumps and collect TRACE (or at least DEBUG) logs from org.infinispan so we can investigate further.
Actions
2. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

pkolanda Jun 19, 2018 1:44 AM (in response to dan.berindei)

Hi Dan,

Thanks for talking time and replying.

We have migrated from infinispan 6.0.2.Final to 9.2.4.Final to solve the split brain merge. Our product is based on the ODL hydrogen code base based. Turning off the merge is not possible as it has lot of data structure to be synced. calling resolveConflicts() on all the data structure will be tidious task.

Below are options we tried out from your reply.
increased thread_pool.max_threads to 32 and increased the lock timeout to 180 secs as the back end operation might take 180 sec for performing the operation. With this the above issue is suppressed for now. Will be keep testing to check this occurs again.

We are hitting the below issue when try to access the empty map. any sugesstions ?
2018-06-13 22:16:14.436 PDT [http-nio-8443-exec-9] ERROR org.infinispan.interceptors.impl.InvocationContextInterceptor - ISPN000136: Error executing command SizeCommand, writing keys []
java.lang.NullPointerException: null
    at org.infinispan.commons.hash.MurmurHash3.hash(MurmurHash3.java:399) ~[infinispan-commons-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.distribution.ch.impl.HashFunctionPartitioner.getSegment(HashFunctionPartitioner.java:45) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.stream.impl.AbstractCacheStream.isPrimaryOwner(AbstractCacheStream.java:425) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.stream.impl.tx.TxDistributedCacheStream.lambda$null$0(TxDistributedCacheStream.java:68) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) ~[na:1.8.0_131]
    at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1620) ~[na:1.8.0_131]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_131]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_131]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_131]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_131]
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_131]
    at org.infinispan.stream.impl.tx.TxDistributedCacheStream.lambda$supplierForSegments$2(TxDistributedCacheStream.java:70) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.stream.impl.termop.SegmentRetryingOperation.performOperation(SegmentRetryingOperation.java:78) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.stream.impl.AbstractCacheStream.performOperationRehashAware(AbstractCacheStream.java:288) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.stream.impl.AbstractCacheStream.performOperation(AbstractCacheStream.java:232) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.stream.impl.DistributedCacheStream.count(DistributedCacheStream.java:426) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:45) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:20) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.impl.CallInterceptor.visitCommand(CallInterceptor.java:29) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.visitSizeCommand(BaseDistributionInterceptor.java:115) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]
    at org.infinispan.interceptors.impl.TxInterceptor.visitSizeCommand(TxInterceptor.java:270) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]
    at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]
    at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:347) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) ~[na:na]
    at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) ~[na:na]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:449) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:444) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:313) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:1082) [na:1.8.0_131]
    at java.util.concurrent.ConcurrentHashMap.<init>(ConcurrentHashMap.java:852) [na:1.8.0_131]
    at org.opendaylight.controller.sal.implementation.internal.FlowProgrammerService.removeFlows(FlowProgrammerService.java:511) [org.opendaylight.controller.sal.implementation-0.5.10-SNAPSHOT.jar:na]
    at com.comp.prod.monitor.internal.Monitor.toggleRuleInstall(Monitor.java:5809) [com.comp.prod.monitor-3.1.9-SNAPSHOT.jar:na]
    at com.comp.one.apps.monitor.web.MonitorWeb.toggleRule(MonitorWeb.java:3200) [com.comp.one.apps.monitor.web-0.6.10-SNAPSHOT.jar:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
    at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219) [org.springframework.org.springframework.web-3.1.3.RELEASE.jar:3.1.3.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) [org.springframework.org.springframework.web-3.1.3.RELEASE.jar:3.1.3.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:100) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:604) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:565) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) [org.springframework.org.springframework.web.servlet-3.1.3.RELEASE.jar:3.1.3.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:755) [equinoxSDK381.javax.servlet-3.0.0.v201112011016.jar:na]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) [equinoxSDK381.javax.servlet-3.0.0.v201112011016.jar:na]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]
    at com.comp.one.apps.monitor.web.cors.CorsFilter.doFilter(CorsFilter.java:65) [com.comp.one.apps.monitor.web-0.6.10-SNAPSHOT.jar:na]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]
    at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:341) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]
    at com.comp.prod.urlredirect.URLRedirectValve.invoke(URLRedirectValve.java:35) [com.comp.prod.urlredirectvalve-0.5.11-SNAPSHOT.jar:na]
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) [orbit.org.apache.catalina-7.0.47.v201312051530.jar:na]
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041) [orbit.org.apache.coyote-7.0.47.v201312060745.jar:na]
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603) [orbit.org.apache.coyote-7.0.47.v201312060745.jar:na]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1721) [orbit.org.apache.coyote-7.0.47.v201312060745.jar:na]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679) [orbit.org.apache.coyote-7.0.47.v201312060745.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

Thanks,Prakash
Actions
3. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

dan.berindei Jun 19, 2018 5:52 AM (in response to pkolanda)

The stack trace implies that there is an entry with a null key in the transaction context (cache.size() counts both entries in the cache and entries modified in the current transaction but not yet committed). AFAIK this should never happen, can you tell us what other cache methods your application calls in the same transaction as cache.size()?
Actions
4. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

dan.berindei Jun 19, 2018 7:57 AM (in response to dan.berindei)

I've created an issue in JIRA: [ISPN-9305] NullPointerException in cache.size() after cache.containsKey()
Actions
5. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

pkolanda Jun 19, 2018 11:41 PM (in response to dan.berindei)

size() is the only operation performed on the empty map. To narrow down.. i stopped all other operation and verified it. Thanks for confirming the bug. That really saves my time.

Is there any safe bed, with partition handing, where we will not hit this issue ?

I am not sure this is related to the same issue. Below execption also happened on the empty map. The operation performed is .values() on the empty map.
2018-06-17 13:00:22.356 PDT [Redirection Interfaces Monitor] ERROR org.infinispan.interceptors.impl.InvocationContextInterceptor - ISPN000136: Error executing command SizeCommand, writing keys []
org.infinispan.util.concurrent.TimeoutException: null
    at org.infinispan.stream.impl.AbstractCacheStream.performOperationRehashAware(AbstractCacheStream.java:313) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.stream.impl.AbstractCacheStream.performOperation(AbstractCacheStream.java:232) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.stream.impl.DistributedCacheStream.count(DistributedCacheStream.java:426) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:45) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:20) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.impl.CallInterceptor.visitCommand(CallInterceptor.java:29) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.visitSizeCommand(BaseDistributionInterceptor.java:115) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]
    at org.infinispan.interceptors.impl.TxInterceptor.visitSizeCommand(TxInterceptor.java:270) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]
    at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]
    at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:347) ~[infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) ~[na:na]
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) ~[na:na]
    at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) ~[na:na]
    at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56) ~[na:na]
    at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitSizeCommand(DDAsyncInterceptor.java:100) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:449) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:444) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.commands.read.AbstractCloseableIteratorCollection.size(AbstractCloseableIteratorCollection.java:47) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at org.infinispan.util.CollectionMapper.size(CollectionMapper.java:34) [infinispan-core-9.2.4.Final.jar:9.2.4.Final]
    at java.util.HashSet.<init>(HashSet.java:118) [na:1.8.0_131]
    at com.comp.prod.monitor.internal.Monitor.redirectionInterfacesMonitor(Monitor.java:8873) [com.comp.prod.monitor-3.1.9-SNAPSHOT.jar:na]
    at com.comp.prod.monitor.internal.Monitor$2.run(Monitor.java:8638) [com.comp.prod.monitor-3.1.9-SNAPSHOT.jar:na]
    at java.util.TimerThread.mainLoop(Timer.java:555) [na:1.8.0_131]
    at java.util.TimerThread.run(Timer.java:505) [na:1.8.0_131]
Actions
6. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

dan.berindei Jun 20, 2018 8:07 AM (in response to pkolanda)

As a workaround for the original NullPointerException, you can change the isolation level to REPEATABLE_READ.

I'm not sure about the TimeoutException, it's very generic so I'd need some information: a thread dump and/or trace logs for org.infinispan.
Actions
7. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

pkolanda Jun 28, 2018 2:43 AM (in response to dan.berindei)

Upgraded to 9.3.0. I get the below error.
[ERROR] Bundle org.opendaylight.controller:clustering.services-implementation:bundle:0.5.11-SNAPSHOT : Classes found in the wrong directory: {META-INF/versions/10/org/infinispan/commons/jdkspecific/ProcessorInfo.class=org.infinispan.commons.jdkspecific.ProcessorInfo, META-INF/versions/10/org/infinispan/commons/jdkspecific/CallerId.class=org.infinispan.commons.jdkspecific.CallerId}

is there a way to come out of it.
Actions
8. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

nadirx Jun 28, 2018 4:19 AM (in response to pkolanda)

You need to upgrade your maven plugins to newer versions that support
multi-release jars
Actions
9. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

pkolanda Jun 28, 2018 4:40 AM (in response to dan.berindei)

REPEATABLE_READ worked. Since the upgrade is not successful i am using this.
Actions
10. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

nadirx Jun 28, 2018 5:32 AM (in response to pkolanda)

https://issues.apache.org/jira/browse/FELIX-5592

You need to use maven-bundle-plugin 3.4.0 or newer
Actions
11. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

pkolanda Jun 28, 2018 11:14 PM (in response to nadirx)

Yes upgrading it to maven-bundle-plugin 3.4.0 with 9.3.0 Final worked.. thanks.. let me test it and update
Actions
12. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

pkolanda Jun 29, 2018 5:53 AM (in response to pkolanda)

Looks like we have to upgrade the osgi. we are using 3.8. Is there a way without upgrading the osgi we make it work ?
Actions
13. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

nadirx Jun 29, 2018 6:27 AM (in response to pkolanda)

You will need to filter out the META-INF/versions/* path from the
infinispan-commons.jar
But I really recommend getting your environment ready for jdk 11, as many
more components will use multi-release jars to ensure backwards/forward
compatibility.
Actions
14. Re: Upgrade from 6.0.2 to 9.2.4- issue after split brain recovery

pkolanda Jul 3, 2018 2:51 AM (in response to nadirx)

What worked for me?
Upgraded to infinispan 9.3.0 Final with the below changes.

We had org.eclipse.osgi-3.8.1.v20120830, this does not support multi-release jars. Updated the maven-bundle-plugin and added <_noee>true</_noee> under configuration so avoid the Require-Capability tag in the Manifest file.
osgi - In bndtools, what causes a Require-Capability header for execution environment to be generated? - Stack Overflow

Thanks for the Infinispan team support.
Actions

1 2 Previous Next

Go to original post