Exception when commiting changes with lucene

[vdepauw]

Hi,

Randomly in modeshape 4.6.0.Final I get the exception below. I'm not sure if the bug is due to modeshape or Lucene and I can't reproduce it in a unit test to prove it.

Since the 5.2.0.Final lot of changes have been made in the related code but I can't migrate right now.

Have you hear about such a problem before that could confirm the nature of the problem?

Caused by: org.modeshape.jcr.index.lucene.LuceneIndexException: Cannot commit index writer
  at org.modeshape.jcr.index.lucene.LuceneIndex.commit(LuceneIndex.java:155) ~[dsdk-launcher.jar:na]
  at org.modeshape.jcr.spi.index.provider.IndexChangeAdapter.completeWorkspaceChanges(IndexChangeAdapter.java:104) ~[dsdk-launcher.jar:na]
  at org.modeshape.jcr.cache.change.ChangeSetAdapter.notify(ChangeSetAdapter.java:157) ~[dsdk-launcher.jar:na]
  at org.modeshape.jcr.spi.index.provider.IndexProvider$AtomicIndex.notify(IndexProvider.java:1493) ~[dsdk-launcher.jar:na]
  at org.modeshape.jcr.bus.RepositoryChangeBus.notify(RepositoryChangeBus.java:190) ~[dsdk-launcher.jar:na]
  at org.modeshape.jcr.cache.document.WorkspaceCache.changed(WorkspaceCache.java:333) ~[dsdk-launcher.jar:na]
  at org.modeshape.jcr.txn.SynchronizedTransactions.updateCache(SynchronizedTransactions.java:223) ~[dsdk-launcher.jar:na]
  at org.modeshape.jcr.cache.document.WritableSessionCache.save(WritableSessionCache.java:751) ~[dsdk-launcher.jar:na]
  at org.modeshape.jcr.JcrSession.save(JcrSession.java:1171) ~[dsdk-launcher.jar:na]
  ... 19 common frames omitted
Caused by: java.nio.file.FileSystemException: C:\Users\Christopher\Infiltrea3CLOUDTEST8\[e-mail]\indexes\default\nodesByPath\_dc_Lucene50_0.doc: The process cannot access the file because it is being used by another process.
  at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) ~[na:1.8.0_92]
  at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[na:1.8.0_92]
  at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[na:1.8.0_92]
  at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115) ~[na:1.8.0_92]
  at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[na:1.8.0_92]
  at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[na:1.8.0_92]
  at org.apache.lucene.util.IOUtils.fsync(IOUtils.java:393) ~[dsdk-launcher.jar:na]
  at org.apache.lucene.store.FSDirectory.fsync(FSDirectory.java:281) ~[dsdk-launcher.jar:na]
  at org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:226) ~[dsdk-launcher.jar:na]
  at org.apache.lucene.store.LockValidatingDirectoryWrapper.sync(LockValidatingDirectoryWrapper.java:62) ~[dsdk-launcher.jar:na]
  at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4456) ~[dsdk-launcher.jar:na]
  at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874) ~[dsdk-launcher.jar:na]
  at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2977) ~[dsdk-launcher.jar:na]
  at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2944) ~[dsdk-launcher.jar:na]
  at org.modeshape.jcr.index.lucene.LuceneIndex.commit(LuceneIndex.java:152) ~[dsdk-launcher.jar:na]
  ... 27 common frames omitted

[hchiorean]

The only thing I can think of is to try configuring the Lucene Index Provider with different lockFactory / directoryClass implementations and check if that helps with this problem. For example if you look at FSDirectory (Lucene 5.3.1 API)  they indicate that using NIOFSDirectory for storing indexes on Windows is bad idea.

[kevin.senechal]

I've post an issue on lucene jira to have the opinion of the lucene dev team. hchiorean can you have a look to the comment on the issue? The one from Mickael McCandless looks interesting.

[LUCENE-7582] "Cannot commit index writer" in some cases on windows - ASF JIRA

Thank you in advance

[hchiorean]

kevin.senechal the ModeShape Lucene index uses one IndexWriter per ModeShape index instance; the storage location for each of the indexes in a given workspace has the form: <basePath>/<workspaceName>/<indexName>, where the first and last segments are coming from the repository configuration.

The IndexWriter is opened for each index, once at repository startup and closed and repository shutdown. In other words, the index writer is kept open throughout the life of the repository, for each index. The flow which writes to the indexes is basically a series of "updateDocument" and "commit" calls, each time something changes for an index (or "remove" and "commit" if you delete data from the index). My understanding of the Lucene docs is that this is the recommended usage pattern for index writers.

Also note that we've tested the Lucene indexes (i.e. ran our pretty extensive JcrQueryManager and LuceneIndexProvider tests) on Windows as well (Windows 7 x64, Oracle JDK1.8.0_66) and they pass consistently. So the issue you are seeing is specific to your use case and/or repository configuration. I would suggesting creating a simple test case which can be run to reproduce this issue.