Thanks for your answer. I made a mistake, you are rigth, index i have seen correspond to index data used by the Infinispan soft-index cache store. It has nothing to do with ModeShape.

When you use local index provider for ModeShape does it mean these data cannot be replicated to other clusters ? I suppose it cannot be.

Thanks, your link helps a lot for the part related to data indexes.

The local index provider (MapDB based) does not support (and as such does not store) FTS. Currently the way FTS works, is if text extraction is enabled, text is extracted and stored from binary values into the binary store. When the query engine encounters FTS criteria, it simply asks the binary stores for the stored text and matches the text against the search criteria.

Not sure to clearly understand what you mean for FTS, binary store and extractors. For example, i use the FileSystemBinaryStore store,  so my binary content (files in my case) is stored into the filesystem. I have also plugged in the tikaExtractor to enable text extraction. What I do not understand is where extraction result is stored. Do you mean tikaExtractor uses the FilesystemBinaryStore used to store binaries configured for binary store ? If I use another binary store, for example DatabaseBinaryStore, does it mean tikaExtractor will also use it ? Basically, does it mean the way text extraction is stored depends on binary store type you use ? Is it possible to separate the way binary is stored and text extraction is stored ? For example, use filesystem to store binary value but use an Infinispan store to store extracted text in order to speed up FTS queries ?

Also, can you please clarify the way FTS queries are working ? For example, when you search for a simple word without any other predicates, does it mean that engine will load one by one all extracted text for all nodes and then search into it or is there any indexes somewhere ?  Does it rely on search engine use by the binary store (for example Windows full text search when files are stored into Windows OS) ? Basically i am wondering what are performances of FTS search when FileSystemBinaryStore is used  and when i will have a lot of binary files nodes. Do you think it would be better to use another kind of binary store for better FTS search performances ?

Regards

Thanks a lot, it is more clear now.

extracted text is always stored in the same place as the binary values. Query-performance wise a local store (FS / ISPN) will be faster than a remote one (a DB across the network).

It is the part I missed, i understand now.

ModeShape 3 used Apache Lucene for performing the full text search queries, while atm. ModeShape 4 uses simple regex matching. When you perform a FTS query, according to the JCR spec, you must specify a selector and either a property name or * meaning all properties. Behind the scenes, if the property is a binary property, ModeShape will ask the binary store for the extracted text of only that binary property and if one doesn't yet exist, it will extract & store that text. The extracted text value is then used against the CONTAINS function.

Yes, i have done more test and analyzed code of FileSystemBinaryStore. I understand better now how binary key is computed and also where extracted text is stored. I can also see extraction triggered and stored upon search. Not very friendly with performances because the 1st user who will access to this specific binary value will face slowdown caused by extraction and storage. It could be interesting to have asynchronous process to automatically extract/store text once binary is saved. Is it something we can configure ? Otherwise I suppose it is something we can do in ModeShape using listeners. If not, i will force extraction with a job querying  asynchronously on new/updated binaries nodes.

Also, text extraction *is not* performed whenever a binary is uploaded but only the 1st time someone queries that binary via FTS (for obvious performance reasons, there is no point in preemptively extracting text and filling a binary store if a user never wants to perform FTS). As such, "true" asynchronous extraction would never work because query results have to be consitent.

I would say the opposite. If users wait for results because the engine need to extract text on first time, it is not really a performance benefit for user's point of view. Basically, asynchronous full text extraction is something usually done for most ECM product i know (ex : EMC Documentum and Full Text server). Full text consistency is not a problem, users usually accept there is a time to wait for data to be ingested into the database. But, i can also understand it is a choice to not overload ModeShape engine, no problem. It is also a consequence of choice to have text extraction done in the same sever. It really depends on application type and if your binary values are often updated or not. For example, if your application is mostly dedicated to read, you expect to have files correctly integrated at the same time as your file is imported into the base. It could be interesting to have the possibility to configure somewhere a list of binary properties we want to extract directly (asynchronously) in the same way as you can configure indexed properties. Anyway, i can deal with that, i will develop my own asynchronous request to force text extraction or use listeners. Thanks.

I have to disagree: full text is nothing more than a feature defined as part of JCR query (via the CONTAINS function) together with all the other JCR-SQL functions. As such, you can't have a query that returns a set of results once and then on subsequent runs another set of results, based on "async stuff" going on in the background, provided none of the repository data - and by that I mean nodes & properties - changes.

I have many years of experiences in EMC Documentum engine that is one of leaders in ECM market and i can guarantee that delay to ingest data is not a problem at all for users. But they are not happy when they need to wait for a search result... In such systems, you have a dedicated server (as dedicated Lucene server for example) to generate full text data and respond to full text requests. Full text indexing are asynchronous and managed by a queue, so during a short laps of time, full text requests do not return your files. It has never been problem and it is commonly accepted.

Anyway, it is not a problem, i can completely understand also ModeShape's choice to extract on demand because in this case the work in done locally and not by a dedicated server. In this case it seems to be the good choice to not overload the server. I think i will use a dedicated ModeShape cluster for extraction work in order to simulate a kind of "fulltext server" and have text extracted/stored before search is done by someone. But i will need to use shared binary store.

Once again, thanks for your great help. Another point of view is always interesting to read.