Hi Sanne,
I've just upgraded to the latest Hibernate Search version as you recommended. Make the search feature in my project work properly is very crucial for me, so I have to try every feasible method.
However, I got the LockObtainFailedException error again with just some index actions (not under load test). The senario is:
- I start 2 local nodes (same configurations)
- Add some objects by node 1, everthing is fine, I can search them on both of the 2 nodes.
- Add one object by node 2, the error is thrown
--------------------
2012-01-02 15:57:01,869 [Hibernate Search: Index updates queue processor for index tc.model.TestCase-1] ERROR org.hibernate.search.exception.impl.LogErrorHandler - HSEARCH000058: Exception occurred org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@21d1cd0d
Primary Failure:
Entity tc.model.TestCase Id 95 Work Type org.hibernate.search.backend.AddLuceneWork
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@21d1cd0d
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1112)
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:125)
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:100)
at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:114)
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:101)
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:69)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
-------------------
- Continue to add objects by node 1, it is still OK.
- Add by node 2, the error occur again.
Please give me some advices on this weird behavior. I might make some configuration mistakes.
1. Jars:
* hibernate-search-engine-4.0.0.Final.jar
* hibernate-search-orm-4.0.0.Final.jar
* hibernate-search-infinispan-4.0.0.Final.jar
* infinispan-core-5.1.0.CR1.jar
* infinispan-lucene-directory-5.0.1.FINAL.jar
* jgroups-3.0.1.Final.jar
2. Spring bean:
<bean id="sessionFactory" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="hibernateProperties">
<props>
<prop key="hibernate.search.default.directory_provider">infinispan</prop>
<prop key="hibernate.search.infinispan.configuration_resourcename">hibernate-search-infinispan.xml</prop>
<prop key="hibernate.search.worker.backend">jgroupsMaster</prop>
<prop key="hibernate.search.worker.execution">async</prop>
....
3. Configuration files:
a. hibernate-search-infinispan.xml
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0 http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<!-- *************************** -->
<!-- System-wide global settings -->
<!-- *************************** -->
<global>
<!-- Duplicate domains are allowed so that multiple deployments with default configuration
of Hibernate Search applications work - if possible it would be better to use JNDI to share
the CacheManager across applications -->
<globalJmxStatistics
enabled="true"
cacheManagerName="HibernateSearch"
allowDuplicateDomains="true" />
<!-- If the transport is omitted, there is no way to create distributed or clustered
caches. There is no added cost to defining a transport but not creating a cache that uses one,
since the transport is created and initialized lazily. -->
<transport
clusterName="HibernateSearch-Infinispan-cluster"
distributedSyncTimeout="300000" >
<!-- Note that the JGroups transport uses sensible defaults if no configuration
property is defined. See the JGroupsTransport javadocs for more flags -->
<properties>
<property name="configurationFile" value="jdbc_ping.xml" />
</properties>
</transport>
<!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.
Hibernate Search takes care to stop the CacheManager so registering is not needed -->
<shutdown
hookBehavior="DONT_REGISTER" />
</global>
<!-- *************************** -->
<!-- Default "template" settings -->
<!-- *************************** -->
<default>
<locking
lockAcquisitionTimeout="300000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false" />
<!-- Invocation batching is required for use with the Lucene Directory -->
<invocationBatching
enabled="true" />
<!-- This element specifies that the cache is clustered. modes supported: distribution
(d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as
with Hibernate Search DirectoryProvider). Replication is recommended for best performance of
Lucene indexes, but make sure you have enough memory to store the index in your heap.
Also distribution scales much better than replication on high number of nodes in the cluster. -->
<clustering
mode="replication">
<!-- Prefer loading all data at startup than later -->
<stateRetrieval
timeout="300000"
logFlushTimeout="300000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="true" />
<!-- Network calls are synchronous by default -->
<sync
replTimeout="300000" />
</clustering>
<jmxStatistics
enabled="true" />
<eviction
maxEntries="-1"
strategy="NONE" />
<expiration
maxIdle="-1" />
</default>
<!-- ******************************************************************************* -->
<!-- Individually configured "named" caches. -->
<!-- -->
<!-- While default configuration happens to be fine with similar settings across the -->
<!-- three caches, they should generally be different in a production environment. -->
<!-- -->
<!-- Current settings could easily lead to OutOfMemory exception as a CacheStore -->
<!-- should be enabled, and maybe distribution is desired. -->
<!-- ******************************************************************************* -->
<!-- *************************************** -->
<!-- Cache to store Lucene's file metadata -->
<!-- *************************************** -->
<namedCache
name="LuceneIndexesMetadata">
<clustering
mode="replication">
<stateRetrieval
fetchInMemoryState="true"
logFlushTimeout="300000" />
<sync
replTimeout="300000" />
</clustering>
</namedCache>
<!-- **************************** -->
<!-- Cache to store Lucene data -->
<!-- **************************** -->
<namedCache
name="LuceneIndexesData">
<clustering
mode="replication">
<stateRetrieval
fetchInMemoryState="true"
logFlushTimeout="300000" />
<sync
replTimeout="300000" />
</clustering>
</namedCache>
<!-- ***************************** -->
<!-- Cache to store Lucene locks -->
<!-- ***************************** -->
<namedCache
name="LuceneIndexesLocking">
<clustering
mode="replication">
<stateRetrieval
fetchInMemoryState="true"
logFlushTimeout="300000" />
<sync
replTimeout="300000" />
</clustering>
</namedCache>
</infinispan>
-----------------------------------
b. jdbc_ping.xml
<config xmlns="urn:org:jgroups" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd">
<TCP bind_port="${jgroups.tcp.port:7800}" loopback="false" recv_buf_size="${tcp.recv_buf_size:20M}"
send_buf_size="${tcp.send_buf_size:640K}"
discard_incompatible_packets="true" max_bundle_size="64K"
max_bundle_timeout="30" enable_bundling="true" use_send_queues="true"
sock_conn_timeout="300" timer_type="new" timer.min_threads="4"
timer.max_threads="10" timer.keep_alive_time="3000"
timer.queue_max_size="500" thread_pool.enabled="true"
thread_pool.min_threads="1" thread_pool.max_threads="10"
thread_pool.keep_alive_time="5000" thread_pool.queue_enabled="false"
thread_pool.queue_max_size="100" thread_pool.rejection_policy="discard"
oob_thread_pool.enabled="true" oob_thread_pool.min_threads="1"
oob_thread_pool.max_threads="8" oob_thread_pool.keep_alive_time="5000"
oob_thread_pool.queue_enabled="false" oob_thread_pool.queue_max_size="100"
oob_thread_pool.rejection_policy="discard" />
<JDBC_PING connection_driver="com.mysql.jdbc.Driver"
connection_username="root" connection_password="root"
connection_url="jdbc:mysql://localhost/clientdb2" level="debug" />
<MERGE2 min_interval="10000" max_interval="30000" />
<FD_SOCK />
<FD timeout="3000" max_tries="3" />
<VERIFY_SUSPECT timeout="1500" />
<BARRIER />
<pbcast.NAKACK use_mcast_xmit="false"
exponential_backoff="500" discard_delivered_msgs="true" />
<UNICAST />
<pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
max_bytes="4M" />
<pbcast.GMS print_local_addr="true" join_timeout="3000"
view_bundling="true" />
<UFC max_credits="2M" min_threshold="0.4" />
<MFC max_credits="2M" min_threshold="0.4" />
<FRAG2 frag_size="60K" />
<pbcast.STATE_TRANSFER />
</config>