Existing index is not returning results with Modeshape 4.5.0, 4.6.2

[javagirlie]

Hello modeshape team,

while trying to upgrade our system from modeshape 4.3.0 to 4.6.2 (still using LocalIndexProvider) we have been experiencing troubles with some of our existing indexes.

The indexes worked well with the previous version (and still works fine with 4.4.0) but are not returning any or only a very limited number of results with 4.5.0 and following versions. 

So far, we have managed to fix one of the indexes which was previously defined this way:

"primaryTypes" : {

  "kind" : "value",

  "provider" : "local",

  "synchronous" : true,

  "nodeType" : "nt:base",

  "columns" : "jcr:primaryType(NAME)"

}

The solution was to change it to the following:

"primaryTypes" : {

  "kind" : "nodetype",

  "provider" : "local",

  "synchronous" : true,

  "nodeType" : "nt:base",

  "columns" : "jcr:primaryType(STRING)"

}

But there remains a problem with another index, which imho has nothing special to it:

"folder" : {

  "kind" : "value",

  "provider" : "local",

  "synchronous" : true,

  "nodeType" : "my:medium",

  "columns" : "folder(LONG)"

}

Here is an excerpt of our node type definitions with some of our own mixins that (hopefully) should not have an impact:

[my:content] > my:baseContent abstract

+ * (nt:base) COPY

- * (UNDEFINED) multiple COPY

- * (UNDEFINED) COPY

[my:medium] > my:authored, my:content, my:versionable abstract

[my:binaryMedium] > my:medium

[my:document] > my:binaryMedium

Originally, we did not have the 'folder' property on our medium node type. But it seems that it doesn't change anything adding it this way:

[my:medium] > my:authored, my:content, my:versionable abstract

- folder (LONG) mandatory COPY

The query still returns hardly any results when the index is considered:

select [jcr:uuid] from [my:medium] where folder = 20

For example: The query returns just one result. Changing the type in the from part to [my:content] (or unregistering the index), so that the index is not considered, leads to a result of 55 entries.

What we have found out is that the index still works with modeshape 4.4.0 but fails with 4.5.0 and upper versions.

We have already tried to reindex the whole workspace as well as rebuilding the index from scratch (by removing the index before repository startup), unfortunately without success.

Do you have any idea what the problem could be or which change in the modeshape versions could cause this problem? We have been debugging the modeshape code but without any success yet.

Thanks in advance,

Susanne

[hchiorean]

It's hard to say outright what the issue could be. Many changes can influence index behavior: bug fixes (or new bugs), the version of MapDB (used for storing indexes) etc. It's really impossible to say since this is the first time (afaik) someone in the community is reporting this type of behavior.

In general, if you remove entirely an index from disk and force a full workspace reindex, nodes from the workspace matching the index definition should appear.

You'll have to debug the differences in behavior locally and see what's going on

[javagirlie]

Thank you for your quick reply Horia. I was worried the answer would be something like that as I know this is (a bit) tricky, especially because I can't provide an example or test yet.

I have removed the entire index and journal from local disk and forced a full workspace reindex via  ((org.modeshape.jcr.api.Session) session).getWorkspace().reindexAsync() but this doesn't change anything, the query does not return the expected results.

Looking at the log while reindexing there are a lot of entries like this:

TRACE [org.modeshape.jcr.index.local.LocalDuplicateIndex] [modeshape-reindexing-5-thread-3] - <Removing node '24ab7a7040f06f652cbbac-d6f7-43e1-8a4b-5abdf4e3b1f1' from 'folder' index with value '20'>

TRACE [org.modeshape.jcr.index.local.LocalDuplicateIndex] [modeshape-reindexing-5-thread-3] - <Adding node '24ab7a7040f06f652cbbac-d6f7-43e1-8a4b-5abdf4e3b1f1' to 'folder' index with value '20'>

TRACE [org.modeshape.jcr.index.local.LocalDuplicateIndex] [modeshape-reindexing-5-thread-3] - <Removing node '24ab7a7040f06f8963eea3-9805-4adb-a382-d343de7ada9e' from 'folder' index with value '20'>

TRACE [org.modeshape.jcr.index.local.LocalDuplicateIndex] [modeshape-reindexing-5-thread-3] - <Adding node '24ab7a7040f06f8963eea3-9805-4adb-a382-d343de7ada9e' to 'folder' index with value '20'>

TRACE [org.modeshape.jcr.index.local.LocalDuplicateIndex] [modeshape-reindexing-5-thread-3] - <Removing node '24ab7a7040f06f15603c45-aa49-462d-ba5d-9b9e9c5e850f' from 'folder' index with value '20'>

TRACE [org.modeshape.jcr.index.local.LocalDuplicateIndex] [modeshape-reindexing-5-thread-3] - <Adding node '24ab7a7040f06f15603c45-aa49-462d-ba5d-9b9e9c5e850f' to 'folder' index with value '20'>

But querying for the nodes having the folder property set to 20 doesn't return these nodes.

Strictly speaking (as I just found out), the query returns a single result which is exactly the node that has been added to the index at last.

{

  "columns": {

    "jcr:uuid": "STRING",

    "jcr:name": "STRING",

    "folder": "LONG"

  },

  "rows": [

    {

      "jcr:uuid": "15603c45-aa49-462d-ba5d-9b9e9c5e850f",

      "jcr:name": "59affa04_c016_4ff3_8543_5c2ec8aeaf6e",

      "folder": "20",

      "mode:uri": "http://localhost:8081/.../59affa04_c016_4ff3_8543_5c2ec8aeaf6e"

    }

  ]

}

Any clues?

[javagirlie]

Hi,

I think, we've identified the location of the issue. There has been a ticket which has been fixed for version 5.0, but it still seems to be a bug in 4.6.2: https://issues.jboss.org/browse/MODE-2576.

We have tried to patch this locally and it works for us. So this (MODE-2576) seems to be an issue not just for boolean indexes but also for those of type long and decimal.

Could you please backport this change to 4.6.2?

If it's necessary we could provide an example and test case.

Thanks in advance,

Bernhard & Susanne

[hchiorean]

I can backport the fix to the 4.x branch, but we will not be doing any more 4.x official releases.

If that helps, let me know and I'll do the backport.

[javagirlie]

Hello Horia.

Thank you for offering to backport the PR. Will it be backported as a bugfix branch branched from the 4.6.2 tag?

[hchiorean]

It would be merged into the 4.x branch. As I said, we're not doing any more 4.x release so you would have to build and use the branch locally.

So the only difference from you building something locally would be that the fix is in the ModeShape repo (as opposed to in a fork you have locally)