Struggling with ContraintViolationExceptions when accessing a node (after write)

[clementp]

I have been trying to get started on ModeShape and I have ran into a problem that I think maybe a race condition when saving child nodes with the same name against a parent node that does NOT allow sns.

Essentially given a CND with:

[foo:baz]

- some properties

[foo:bar]

+ * (foo:baz)

If one were to write using two threads concurrently to a node of type foo:bar with the name "node1". Those two writes would succeed and subsequent reads to that node would fail with a CVE. Looking at the code that actually determines the child definition of a node I believe confirms my theory. In the method "findChildNodeDefinition" within RepositoryNodeTypeManager, it uses the numberOfExistingChildrenWithSameName as the search parameter when determining the child node definition. As such, if two childs are sucessfully written concurrenlty (I would have expect one write to fail with some sort of a conflict but none was observed), the child definition would be "lost". The code works right away if foo:bar is instead replaced by nt:unstructured

Obviously in that case, you wouldn't be able to detect and reject duplicate child nodes (since unstructured allows sns).

I am trying to concoct a test case to demonstrate this but I just wanted to see if the devs think. In particular, should one of the writes fail (does it fail silently)? What is the expected behavior in a case like this? Should the developer "lock" the parent node before attempting the write to fix this (although I think this is a rather common use case of transactions and isolation)?

For the record, I was using ModeShape 3.1.2.Final with Jboss 7.1.1.Final, Infinispan 5.1.2 with PESSIMISTIC locking in FULL_XA mode. The error message is:

javax.jcr.nodetype.ConstraintViolationException: Cannot find a definition for a child named 'node1' on the node at '/documents' with primary type 'foo:bar' and mixin types: []

[rhauch]

If one were to write using two threads concurrently to a node of type foo:bar with the name "node1". Those two writes would succeed

This actually is not the way ModeShape works. ModeShape uses node-level transactional locks, which means even if the two sessions are saved at exactly the same time, only one session will obtain the lock in its transaction, forcing the other session (with a different transaction) to wait until the lock is released (when the other transaction commits or rolls back). We tried to explain that a bit in this blog post.

Now, it is a possibliity that the two sessions are timed such that they both pass their validation but then lock on the persisting operation.

Can you create a simple test case that replicates this? And can you please log an issue in our JIRA?

In particular, should one of the writes fail (does it fail silently)? What is the expected behavior in a case like this?

It does not fail at the moment, so it is probably persisting both the child nodes. When the second one is read, it is likely encountering the inability to find a child node definition for the second child node. Does this match where you're seeing the exception being thrown?

Should the developer "lock" the parent node before attempting the write to fix this (although I think this is a rather common use case of transactions and isolation)?

That certainly would be a workaround to prevent the concurrent sessions from modifying the same parent node, though it's not ideal.

[rhauch]

Oh, one more thing. It shouldn't be related to AS7, and should be reproducible in a JavaSE environment with a real transaction manager. Note that our unit tests fit the bill, and you could probably add a test method to ConcurrentWriteTest. In particular, this existing test is relatively similar (though it's simultaneously removing rather than adding) and uses a CyclicBarrier to artificially force the session.save() calls to occur at the same time (or as close as we can get them):

     https://github.com/ModeShape/modeshape/blob/master/modeshape-jcr/src/test/java/org/modeshape/jcr/ConcurrentWriteTest.java#L161

If you don't want to create a test case, just having your client code and complete CNDs would be sufficient.

[clementp]

Thanks Randall. I'll have the code checked out and tests running inside IntelliJ already, I am just trying to understand the CND that's being used (since I need a custom one that's not nt:unstructured) and how the test harness is used.

[clementp]

Issue created in Jira: https://issues.jboss.org/browse/MODE-1821

Testcase created in Github: https://github.com/panghy/modeshape/commit/4d268ee8369a052fc329e72dc9c4ac33e258e211

Let me know if more background info is needed. The test case shows that serliazed call do fail but concurrent calls succeed with more than the expected number of nodes written (no failures on the save).

[clementp]

Fixed in 3.1.3.Final. Thanks.