Infinispan with Lucene 4.x

[ges]

What do I need to do to configure ISPN 6.0 to use Lucene 4.x ? It looks like ISPN 6 supports lucene 4.x but the default maven setup is to use Lucene 3. I tried to add infinispan-lucene-v4 as a maven dependency but that will not work as it only brings in a subset of dependencies.

Is there some configuration that I need to add to infinispan.xml? And what about maven dependency?

Thanks,

Gesly

[sannegrinovero]

Hi,

you just need to use <groupId>org.infinispan</groupId><artifactId>infinispan-lucene-directory</artifactId> in your application.

The split in -v3 and -v4 is something we only need at build time.

To get a Directory instance make sure you don't use the (deprecated) constructor but use the builder at org.infinispan.lucene.directory.DirectoryBuilder: this will make sure you'll get the correct implementation, depending on if it can detected Lucene 3 or 4 in the classpath.

Infinispan 7 will support just Lucene 4, we needed this trick to be temporarily in place to alllow for a migration path.

[ges]

Saane,

infinispan-query brings in lucene 3 as a dependency and I'm trying to override that to use lucene 4 - not clear to me how to get the combination of hibernate search and lucene 4 dependencies.

For defining the version, can I do something similar to this in infinispan.xml - this is from the remote query quick start project.

	<!-- Enable indexing using the RAM Lucene directory provider -->
	<indexing index="ALL">
	<property name="default.directory_provider">ram</property>
	<property name="lucene_version">LUCENE_36</property>
	</indexing>

What is on the roadmap for infinispan 7? It that posted somewhere?

[ges]

I tried the following combination with hibernate 5 alpha1 but still getting run time errors.

	<dependency>
	<groupId>org.infinispan</groupId>
	<artifactId>infinispan-core</artifactId>
	<version>6.0.1.Final</version>
	</dependency>
	<dependency>
	<groupId>org.infinispan</groupId>
	<artifactId>infinispan-query</artifactId>
	<version>6.0.1.Final</version>
	</dependency>
	<dependency>
	<groupId>org.apache.lucene</groupId>
	<artifactId>lucene-core</artifactId>
	<version>4.6.1</version>
	</dependency>
	<dependency>
	<groupId>org.hibernate</groupId>
	<artifactId>hibernate-search-infinispan</artifactId>
	<version>5.0.0.Alpha1</version>
	</dependency>
	<dependency>
	<groupId>org.hibernate</groupId>
	<artifactId>hibernate-search-engine</artifactId>
	<version>5.0.0.Alpha1</version>
	</dependency>
	<dependency>
	<groupId>org.apache.lucene</groupId>
	<artifactId>lucene-analyzers-common</artifactId>
	<version>4.6.1</version>
	</dependency>

This is error message I get with the above combination of jars,

java.lang.NoClassDefFoundError: org/hibernate/search/infinispan/CacheManagerServiceProvider

    at org.infinispan.query.backend.SearchableCacheConfiguration.initializeProvidedServices(SearchableCacheConfiguration.java:72)

    at org.infinispan.query.backend.SearchableCacheConfiguration.<init>(SearchableCacheConfiguration.java:41)

    at org.infinispan.query.impl.LifecycleManager.getSearchFactory(LifecycleManager.java:212)

    at org.infinispan.query.impl.LifecycleManager.cacheStarting(LifecycleManager.java:73)

    at org.infinispan.factories.ComponentRegistry.notifyCacheStarting(ComponentRegistry.java:228)

    at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:214)

    at org.infinispan.CacheImpl.start(CacheImpl.java:675)

    at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:553)

    at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:516)

    at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:398)

    at com.dci.compliance.service.CacheManager.getCache(CacheManager.java:27)

[sannegrinovero]

Hi, I initially thought you where asking about the Lucene Directory to use with Lucene 4: that works with the dependency as I've posted above.

But now  that you mention infinispan-query : I should have immediately clarified that the Query capability strictly requires Apache Lucene 3.6.2.

It will use Lucene 4 in Infinispan 7, but master isn't ready to be migrated to Hibernate Search 5 yet.. will take a couple of weeks still.

If you need to use Infinispan Query today I'd suggest to use Lucene 3.6. If you need strictly Lucene 4, could you please share your reasons?

[ges]

Hi Saane,

Sorry about the confusion. I guess I haven't thought of lucene with Infinispan outside the context of query capabilities. I forgot that you can use Infinispan as the store for lucene indexes. I was trying lucene 4 as I had seen one of your threads about lucene 4 & Infinispan performance improvements. I was trying to see if indexing (for querying the cache) performance would improve on switching to Lucene 4.

I can live with Lucene 3 for now.

Is the road map for Infinispan 7 posted somewhere?

[sannegrinovero]

Hi, yes the Roadmap used to be online, but the one I've found is outdated.. I've asked on the mailing list to see if that can be fixed.

Anyway, in the big picture of important changes, you'd probably not have found the Lucene 3/4 update on the roadmap as it's an implementation detail.

Note that if you have performance trouble with Lucene 3, there are lots of tuning options you can play with for very significant benefits. These can't be enabled out of the box as sadly there is no one size fitting for all, tuning it requires a bit of understanding of your use case.

This could be useful: Hibernate Search Tuning chapter.

[ges]

sannegrinovero

I hit upon performance issues every now and then with Lucene indexing in Infinispan. This is the scenario I have.

I've 2 threads that updates a local cache - 1 thread add about 15k entries and another thread add about 2.9 million entries. Initially, both threads are adding new entries to the cache. Later, these threads are replacing existing entries in the cache. It is during this phase that I hit upon performance issues.

My cache is configured as follows:

	<namedCache name="dataSource">
	<indexing enabled="true" indexLocalOnly="true">
	<properties>
	<property name="default.indexmanager" value="near-real-time"/>
	<property name="default.directory_provider" value="infinispan"/>
	<property name="default.worker.thread_pool.size" value="16"/>
	<property name="default.worker.execution" value="async"/>
	<property name="default.indexwriter.ram_buffer_size" value="1024"/>
	<property name="default.max_queue_length" value="50000"/>
	<property name="default.indexwriter.merge_factor" value="30000"/>
	</properties>
	</indexing>
	</namedCache>

Entries are added to the cache using the putAll(Map entries) method. When 2.9 million entries are replaced, the putAll method takes a really really long time to return (I've seen cases where it takes about 108 minutes to return). The value added to the cache has 6 fields marked for indexing.

When fresh entries are being added to the cache, the putAll takes about 4 minutes to return.

Am I doing something wrong here? Is there I can do to improve this? Is there some way to batch the putAll() op so that indexing is done after all the new entries are added or enable async indexing?

Appreciate all your help with this.

Thanks,

[ges]

sannegrinovero Any suggestions?