-
1. Re: Clustering - sync/async replication
rhauch May 1, 2014 9:58 AM (in response to eshevchenko)1 of 1 people found this helpfulI do NOT think it is wise to use ModeShape with async replication (e.g., how Infinispan transfers state and coordinates state activities). ModeShape is strongly-consistent, and (unlike a lot of other Infinispan applications) requires consistent views of related entries/nodes. Using async replication means that ISPN is no longer required to provide ModeShape a strongly-consistent view of the cached information. I think the problems you listed above are just the tip of the iceberg of possible side effects.
But just to clarify, sync/async replication is very different from sync/async cache store writes (e.g., write-through or write-behind). I do think it's possible to use write-behind with ModeShape, as the Infinispan cluster still stays in sync via state transfer at the time of writes, and write-behind just means that the cache store is updated asynchronously.
-
2. Re: Clustering - sync/async replication
eshevchenko May 5, 2014 12:12 PM (in response to rhauch)Randall, thanks for your answer.
As for async replication. I have a Webservices bridge before Modeshape and planed to use load balancer with configured sticky sessions to route the requests to a particular node. So the replication will be finished by the time client is navigated to another node. But if you do not recomend to use async replication with Modeshape at all I'll stay with sync mode for replication.
As for sync replication. Test with "moving nodes" is unstable for sync replication as well.
I run this test with 4,10,15,20 threads and have different results after each launch. Usually one or two tests are failed with exception like this
Caused by: org.junit.internal.runners.model.MultipleFailureException: There were 2 errors:
java.util.concurrent.ExecutionException(javax.jcr.InvalidItemStateException: This session tried to save changes to node with key 'Cannot locate child node: 4c24b267505d646d397f81-a9be-445d-a3be-77ec166f725e within parent: 4c24b267505d644e5bd1c1-6a7d-46fb-a0c1-9ea4955074a9', but it was removed by another session.)
When I run tests with JProfiler - all tests are passed.
I can't figure out, is it a bug, or configuration issue?
-
3. Re: Clustering - sync/async replication
rhauch May 7, 2014 11:57 AM (in response to eshevchenko)It may be a configuration issue if your repositories are not properly clustered. For example, with 3.x you have to make sure that both ModeShape and Infinispan are clustered properly, and that all have access to the same shared cache store. If so, and you're still seeing a problem with move, then please provide a test case with configuration information that demonstrates it -- we'll be happy to look into it.
Some of it also depends on what your move tests are doing. For example, if your test is trying to concurrently move the same nodes in separate sessions, then you should expect that the first session to complete this will succeed, and that all subsequent sessions will fail with an exception. After all, we only reserve locks for writing nodes during the save operations; locking nodes while the session is making all the individual changes would dramatically reduce performance, and could be disastrous for normal concurrency.
-
4. Re: Clustering - sync/async replication
eshevchenko May 15, 2014 11:40 AM (in response to rhauch)Thanks, hint with shared cache store add more stability. My previous configuration has own cache storage for each cluster node. The link to testcase that was in my previous message is the same - https://github.com/eshevchenko/modeshape/blob/cluster-benchmark/modeshape-jcr/src/test/java/org/modeshape/jcr/benchmark/MoveDocumentsBenchmarkTest.java.
Main concern : I have an initial content file. It contain source folder and destination. The test has read the source folder and create callable task for each jcr node. The task will move this node to destination folder. So the case " if your test is trying to concurrently move the same nodes in separate sessions" shouldn't be happens. Test is running with different count of threads:
moveDocumentsSyncTcp - number of treads are the maximum number of processors available to the virtual machine
moveDocumentsSyncTcp10 - number of treads are 10
moveDocumentsSyncTcp15 - number of treads are 15
moveDocumentsSyncTcp20 - number of treads are 20
Each test is running for 3 execution(BenchmarkOptions - benchmarkRounds)
The count of cluster nodes is 2
After each execution i have different results. Increasing cluster node count has lead to increasing errors count. Increasing the count of benchmarkRounds has lead to increasing errors count. Error is still the same
java.lang.reflect.InvocationTargetException
at com.carrotsearch.junitbenchmarks.BenchmarkStatement$BaseEvaluator.evaluateInternally(BenchmarkStatement.java:90)
at com.carrotsearch.junitbenchmarks.BenchmarkStatement$SequentialEvaluator.evaluate(BenchmarkStatement.java:121)
at com.carrotsearch.junitbenchmarks.BenchmarkStatement.evaluate(BenchmarkStatement.java:346)
at org.modeshape.jcr.benchmark.AbstractBenchmarkTest$1$1.evaluate(AbstractBenchmarkTest.java:132)
at org.junit.rules.RunRules.evaluate(RunRules.java:18)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:77)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:195)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: java.util.concurrent.ExecutionException: javax.jcr.InvalidItemStateException: This session tried to save changes to node with key 'Cannot locate child node: 4c24b267505d649a2c9bcd-b65b-494b-a15d-e791d40f6ac4 within parent: 4c24b267505d64df01b090-ef0d-4e4e-b80a-b52a832c988a', but it was removed by another session.
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.modeshape.jcr.benchmark.AbstractBenchmarkTest.executeTest(AbstractBenchmarkTest.java:203)
at org.modeshape.jcr.benchmark.MoveDocumentsBenchmarkTest.moveDocumentsSyncTcp20(MoveDocumentsBenchmarkTest.java:116)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at com.carrotsearch.junitbenchmarks.BenchmarkStatement$BaseEvaluator.evaluateInternally(BenchmarkStatement.java:76)
... 22 more
Caused by: javax.jcr.InvalidItemStateException: This session tried to save changes to node with key 'Cannot locate child node: 4c24b267505d649a2c9bcd-b65b-494b-a15d-e791d40f6ac4 within parent: 4c24b267505d64df01b090-ef0d-4e4e-b80a-b52a832c988a', but it was removed by another session.
at org.modeshape.jcr.JcrSession.save(JcrSession.java:1152)
at org.modeshape.jcr.benchmark.MoveDocumentsBenchmarkTest$MoveNodeTask.call(MoveDocumentsBenchmarkTest.java:221)
at org.modeshape.jcr.benchmark.MoveDocumentsBenchmarkTest$MoveNodeTask.call(MoveDocumentsBenchmarkTest.java:186)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.modeshape.jcr.cache.NodeNotFoundInParentException: Cannot locate child node: 4c24b267505d649a2c9bcd-b65b-494b-a15d-e791d40f6ac4 within parent: 4c24b267505d64df01b090-ef0d-4e4e-b80a-b52a832c988a
at org.modeshape.jcr.cache.document.LazyCachedNode.parentReferenceToSelf(LazyCachedNode.java:250)
at org.modeshape.jcr.cache.document.LazyCachedNode.getSegment(LazyCachedNode.java:287)
at org.modeshape.jcr.cache.document.LazyCachedNode.getPath(LazyCachedNode.java:313)
at org.modeshape.jcr.cache.PathCache.getPath(PathCache.java:49)
at org.modeshape.jcr.cache.document.WritableSessionCache.persistChanges(WritableSessionCache.java:994)
at org.modeshape.jcr.cache.document.WritableSessionCache.save(WritableSessionCache.java:621)
at org.modeshape.jcr.JcrSession.save(JcrSession.java:1145)
Let me know if this is a topic for another discussion
Thanks by advance.
-
5. Re: Clustering - sync/async replication
eshevchenko May 21, 2014 12:08 PM (in response to rhauch)Randall, I reworked test case, i think this one should give you better understanding what i mean. Appropriate pull request was created - [MODE-2216] Move operation on cluster causes InvalidItemStateException exception - JBoss Issue Tracker
Thanks by advance.