Fix missing child references

[wesssel]

Hey everyone,

We have experienced some inconsistency in our Modeshape repository. The cause is unknown, it might be related to OOM exceptions caused by MODE-2466, MODE-2280 (ISPN-4810) or ISPN-201, that is not important right now.

The exception we are getting is the following:

nested exception is javax.jcr.PathNotFoundException: No item exists at path <path> relative to <parent path> in workspace "default"

Which occurs when we are calling a Node.getNode(path) on Node <parent path>. This method is called right after we are trying <parent path>.add(path), which throws an ItemExistsException. This was a workaround for indexes that are not accurate, caused by Modeshape nodes periodically leaving the cluster, caused by OOMs since there was a workspace cache without eviction.

try {

    child = node.addNode(id, "nt:folder");

} catch (ItemExistsException e) {

    node.getNode(id).remove();

    child = node.addNode(id, "nt:folder");

}

Is there any way to programatically fix this ourselves with Modeshape's JCR api or would we need to delve deeper into Modeshape? Modeshape version is 3.x

[hchiorean]

There isn't anything you can do with IDs since they are completely internal and auto-generated by ModeShape. You cannot add a node with an explicit id (it's outside the JCR spec).

If your use case allows it, you can try getting the "missing" node by ID, copying its data into a new node, removing the old node and adding the new node under the same parent.

.This was a workaround for indexes that are not accurate, caused by Modeshape nodes periodically leaving the cluster, caused by OOMs since there was a workspace cache without eviction.

You should be aware that ModeShape (and Infinispan for that matter) cannot handle network partitions and remain consistent, so if you're experiencing this you'll have to come up with a solution yourselves. Being a JCR spec implementation, ModeShape requires strong consistency (at least linearizability) and this can't be guaranteed by ISPN in the presence of partitions.

[wesssel]

Removal of the old node will not work, since the node cannot be found under its parent, causing exceptions. Network partitions was not the problem we were experiencing. Parts of the Modeshape cluster just crashed altogether, leaving them with inaccurate indexes as nodes were added on the cluster that was still running. Thus when trying to add a node after querying for its existence on a restarted instance, the query can come up false. When trying to add, the node did however already exist under the parent, which is why we implemented the try-catch block.

I guess this is a case that could be fixed when implementing https://issues.jboss.org/browse/MODE-1641

[hchiorean]

I guess this is a case that could be fixed when implementing https://issues.jboss.org/browse/MODE-1641

IMO this issue is more "wishful thinking" than something with a realistic chance of implementation. JCR-2.0 *is hugely complex* meaning it's highly unlikely to figure out what "repairing a repository" means. Moreover, data corruption can mean more inconsistencies than just in terms of the JCR2.0 invariants, but also in terms of internal data structures (the problem you're describing is such an example). ModeShape is essentially a JCR2.0 implementation, not a data store/database (for which such a utility would make more sense perhaps).

The more realistic approach to data corruption is tracking down the cause of the corruption (via samples/test cases) and trying to solve it so that it doesn't happen again. Issues like [MODE-2420] Modeshape can potentially lose data because Infinispan Cache Stores do not participate in transactions - JBo…  on the other hand mean that in theory at least, there can be cases of data corruption/inconsistency where ModeShape can do absolutely nothing as they are caused by Infinispan's design.