1 of 1 people found this helpful
I've run some tests on the versions you mentioned, and am going to add some automated to trunk soon to cover for such situations in future, but so far I've seen no issue.
I'm testing that removing from or resetting the index correctly resets the index state, or shrinks it.
I think this issue might be related to LUCENE-3147 : if Lucene fails to close a segment or chunk reference, we will consider it still in use, and so we'll prevent it from being deleted as some open Searcher instances might need the next chunk. Could you try keeping the same version of Hibernate Search but update Lucene to at least 3.2 ?
Luckily while Lucene changes API very frequently, 3.2 and 3.1 are both compatible with Search 3.4.1
I'll try that tomorrow, that upgrade should be simple!
I'm upgrading Lucene rigth now.
About LUCENE-3147 I understand it only affects a test component "MockDirectoryWrapper", doesn't it?
That's correct, but while improving the test utility as you can see in the comments they have found several unclosed file handles, and fixed them as part of the same issue.
While I was upgrading Lucene I've found an improperly closed IndexReader in the application code. Maybe that has something to do with those file handles. I'll fix that and test my application again.
great, that would explain a lot
I hope it's clear on why it avoids deleting index chunks while they are open; this would not be required if you could live without chunking: I'm considering for the next release of Hibernate Search to configure (by default) the IndexWriter to not create big segments anyway, and make sure the chunking threshold is higher than the expected segment size. One should still avoid defining a too high chunksize buffer as it also affects how big the buffers are allocated internally.
Those prx "files" are still there in my cache after fixing that IndexReader leak.
I've just finished the Lucene upgrade, now I am running my app with Lucene 3.2.0.
Tomorrow will see how it works.
Also check to make sure you have no other IndexReader leaks, make sure you always close them in a finally block right after opening them.
In worst case, we can see how you could reset the index state periodically, if that's an option.
Unfortunatelly the cachestore row count keeps growing after the upgrade to 3.2.0.
When I run an index optimization the size grows even more!
Is there a way to find out what could be locking the indexes? Or maybe a way to force the release of the locks and optimize after that.
After some days trying to fix this without any success, I've found that disabling all the Hibernate/Lucene automatic optimization prevents the crazy index growth.
I removed this entries from my persistence.xml file:
Now my index grows slowly, as it should.
Maybe I'll have performance issues later because of the lack of index optimization, and would be forced to do a full reindex from time to time.
thanks for coming back with this information. It seems likely there is a file leak triggered by the IndexWriter optimize() .. you might want to look there or at least let the Lucene mailing list know about your findings.
Your workaround looks fine: optimizing an interactive index (one on which the application writes frequently) is quite pointless and Lucene is pretty good nowadays even on non-optimized indexes making it less of a requirement to actually do optimization. It's entirely possible that for your use case it was a better option anyway to disable this.