[Modeshape 5.2] Problem in indexing configuration and persistence

[l.tagliani]

Hi all,

  we are testing the indexing feature of Modeshape to fasten our searches.

In particular, we are testing the creation of indexes at runtime, because we have use cases in which the indexes can be added dynamically.

But we encounter a strange behaviour: it seems that the indexes are lost between restart of the system.

The configuration seems to be correctly persisted, but when we restart the repository, indexes are not recognized; this is verified using IndexManager.getIndexDefinitions().

This happens with all of three index providers: local, Lucene, Elasticsearch.

I've created a test suite to simplify the detection of any error (or our misunderstanding) at https://github.com/lucapino/modeshape-indexes.

These tests demonstrate that if we first create the indexes they are immediately recognized and the query will use it.

After we restart the system, the indexes are not found and used by the query system.

Perhaps we are making some mistakes that's not easy to spot, but we can't find one.

Are we doing things correctly?

BR

Luca Tagliani

P.S.: in addition to this, during th use of Elasticsearch index provider, the Elasticsearch server continues to log a WARN message like this (the exception message is in italian ;-) ):

... - *WARN * [elasticsearch[Iron Man][http_server_worker][T#3]{New I/O worker #21}] netty: [Iron Man] Caught exception while handling client http traffic, closing connection [id: 0xe93df166, /127.0.0.1:54562 => /127.0.0.1:9200] (exceptionCaught in NettyHttpServerTransport.java, line 363) java.io.IOException: Connessione in corso interrotta forzatamente dall'host remoto

Is it a normal behaviour of the system? In fact the indexing (not searching) performance with Elasticsearch are not as good as we expected, compared to Lucene or local indexing.

[hchiorean]

There is most likely a bug around the IndexManager API, so please open a JIRA for this (the indexes are being persisted but they are not recognized and disabled at startup). Until the bug is fixed, you should be able to define the indexes statically in the JSON file and they should work as expected between restarts.

As far as the ES warning you'll have to investigate locally what the network issue (IOException) is.

[l.tagliani]

Hi Horia,

     I've opened MODE-2644 for this bug.

I've another question regarding use of static index definition.

We are creating namespaces and nodetype at runtime, and the use of initial content feature is not suitable for us.

If we define statically those index, the repository doesn't start stating that one of our custom namespace is not defined (which is correct!).

There's an alternative way for doing this?

Otherwise we cannot use the indexing feature at all, meanwhile the issue is fixed.

Also, perhaps, trying to use your suggestion I've probably found another issue....

Only using Elasticsearch index provider, upon each engine restart all the indexes in Elasticsearch are removed and recreated, even if statically defined in JSON and even if they already exists.

I think that this can be related to MODE-2644, because browsing IndexManager code, it seems that the existence of index is not correctly detected by the notify method at line 969-970.

Should I open another issue for that?

BR

Luca Tagliani

[hchiorean]

If you're dynamically registering your types, then no, you won't be able to make proper use of indexes until MODE-2644 is fixed.

Regarding ES, I don't understand what part of the code you're referring to exactly, but you should keep in mind that if there's an ES provider bug which doesn't manifest for all the other providers, then the bug can only be in the ES provider codebase (which is a separate artifact). The indexing SPI found in modeshape-core is shared by all index providers, so bugs there should show up the same for all the providers.

[l.tagliani]

Hi Horia,

     you are right, the problem arise in the elasticsearch-index-provider.

In fact, differently from the Lucene provider, when the system starts and the core indexManager tells to the provider to create indexes, the ES provider always drop the index if it exist, then creates it.

This could be right if the repository is new, but if the repository is not new, the index shouldn't be recreated, otherwise once the MODE-2644 is fixed, there will be a full reindexing every restart of the engine.

If you think it's right, I will create a new issue for this (to me) incorrect behaviour of the ES index provider.

BR

Luca Tagliani

[hchiorean]

Not sure why the ES index provider is deleting indexes each time it starts up, so yes, please open a JIRA; thanks