Search batch indexing

[john.prince]

Hi,

Just a quick question. I have been playing with the query api, which is very cool, but seems very slow when adding large numbers of items to the cache.

I have a test which adds a couple of hundred thousand entries, and indexing is a non-starter. Is there, or will there be, an option to disable indexing while adding lots of entries and then index them all at once? I am thinking of the hibernate search FullTextSession batch features which seem to offer this (I don't know how well they perform, because I haven't used hibernate search before, but it looks like they were designed for this).

Best wishes

John

[imbng]

I had a similar problem with slow loads when using an index. I'm curious to hear and answer from the devs but if for me I wrapped the preload in a transaction and made things go much faster.

Bryan

[john.prince]

An update - using batching for the caching process helps a lot. Is there anything else I could do?

Async indexing might make sense for quicker startup (though it seems slower - it doesn't seem use all the available cpu time, for some reason) - but is there any reliable way I can find out if the indexing is finished?

[manik]

Interesting. You guys should bring this up on the Infinispan dev mail list. Remember that querying is a tech preview at the moment and not finalised. Both batching (on a Hibernate Search level) and async indexing would make a lot of sense.