Hibernate Search master/slave setup with Infinispan

[ekfliu]

I have a master/slave jms hibernate search setup running using NFS and is working correctly. Unfornately with nfs locking issues with lucene, eventually the index becomes corrupt and requires reindex or recopying of the index from source directory.

I have attempted to convert the current setup to infinispan directory but only with partial success. I still want to follow the original master/slave setup the only difference is directory provider. Unfortnately it looks like the IndexReader cannot read any data from the cache and returns nothing. I can see the cache store file directory getting filled up, jmx-console also indicate data in the cache, but the reader cannot see any index. Rebooting the machine causes the provider to load the index from file system and everything works fine, but any subsequent index does not show up. I have setup hibernate search to use no share index reader(causes new index reader to be loaded each time), but it made no difference.

I have temporary disable the master/slave setup and uses regular hibernate search setup, every thing works correctly, search show data as soon as index start.

So the question is does it still make sense to use master/slave approach for infinispan directory setup? what causes infinispan to not show data to the other client? The index is done in a MDB in a ejb-jar file while the reader is from a war file, both are setup to use jta transaction. How does infinispan knows when to show the data to other client? I assume its using transaction but it does not work when clients are different. It works fine when both index is done in the same war file.

[sannegrinovero]

Hi,

I still want to follow the original master/slave setup the only difference is directory provider.

Yes that's how it's intended to be used.

Unfortnately it looks like the IndexReader cannot read any data from the cache and returns nothing. I can see the cache store file directory getting filled up, jmx-console also indicate data in the cache, but the reader cannot see any index. Rebooting the machine causes the provider to load the index from file system and everything works fine, but any subsequent index does not show up.

It looks like your nodes are not connected in cluster. You should enable logging at JGroups and/or Infinispan level to verify that each node actually "sees" each other. It might also help to start the Master node first, so that there's no doubt about which node is going to initialize the index (shouldn't be needed, but I'll start this way which is simpler).

Also, please show your configuration files, I didn't understand which cache store you're using, or how your IndexReaders are able to find the index after a restart.

I have setup hibernate search to use no share index reader(causes new index reader to be loaded each time), but it made no difference.

Both options should work, but shared readers are known to provide better performance so they are recommended. Until we fix the main issue however you can keep the non-shared, at least we rule out some extra complexity.

I have temporary disable the master/slave setup and uses regular hibernate search setup, every thing works correctly, search show data as soon as index start.

So the question is does it still make sense to use master/slave approach for infinispan directory setup? what causes infinispan to not show data to the other client? The index is done in a MDB in a ejb-jar file while the reader is from a war file, both are setup to use jta transaction.

Yes it makes still sense to use the master/slave approach, especially if you have very frequent writes. If a write operation is not very frequent you could keep the current configuration, but the risk with this configuration is that multiple nodes might want to try writing on the index at the same time; they will have to acquire the Directory lock before the write, and if some other node has it and keeps the lock for a long time the other nodes might get a timeout exception. When using the master/slave approach instead, you can configure the master node to use exclusive_index, which means he will always keep the IndexWriter lock for himself and be more efficient in writes, so you have no timeouts to fear.

How does infinispan knows when to show the data to other client? I assume its using transaction but it does not work when clients are different. It works fine when both index is done in the same war file.

This is the sequence:

- Hibernate Search queues update operations in the transactional context (local to your node)

- At transaction commit, Hibernate Search applies all changes to the index, writing into Infinispan

- Infinispan writes the index updates to the other nodes immediately (depending on your configuration)

- Each other node check the index state to know if they need to refresh their IndexReader(s) pool at the beginning of each transaction (actually when each transaction for the first time asks for an IndexReader), and reuses the same until the transaction terminates (effectively providing "repeatable read" isolation on the index).

In short, other nodes don't need to be notified when a remotely executing transaction is committed.

[ekfliu]

Thanks for the response, currently I am just testing it under one server right now locally, so no clustering is involved right now. The only difference between the working version and the non working version is indexing writing and reading are done separately with an ejb and war application, while the working version does both in the war application.

Unless you are saying propagation of infinispan data also work like clustering between the ejb and war application on the same server.

I am using the infinispan configuration on the jboss 6 server and not a standalone hibernate search infinispan configuration, below is the configuration.

{code:xml}
<!-- hibernate search index definitions -->
  <infinispan-config name="hibernateSearch" jndi-name="java:CacheManager/searchIndex">
    <alias>hibernate-search-cache</alias>
    <infinispan xmlns="urn:infinispan:config:4.2">
      <global>
        <transport clusterName="${jboss.partition.name:DefaultPartition}-Search" distributedSyncTimeout="17500">
          <properties>
            <property name="stack" value="${jboss.default.jgroups.stack:udp}"/>
          </properties>
        </transport>
        <globalJmxStatistics enabled="true"/>
        <shutdown hookBehavior="DONT_REGISTER"/>
      </global>
      <default>
        <locking lockAcquisitionTimeout="20000" writeSkewCheck="false" concurrencyLevel="500" useLockStriping="false" />
        <jmxStatistics enabled="true"/>
        <lazyDeserialization enabled="false" />
        <invocationBatching enabled="true"/>
        <eviction maxEntries="-1" strategy="NONE" />
    <expiration maxIdle="-1" />
        <clustering mode="replication">
           <!-- Prefer loading all data at startup than later -->
           <stateRetrieval timeout="20000" logFlushTimeout="30000" fetchInMemoryState="true" alwaysProvideInMemoryState="true" />    
           <!-- Network calls are synchronous by default -->
           <sync replTimeout="20000" />
        </clustering>
        <loaders passivation="false" shared="false" preload="true">
          <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true" purgeOnStartup="false">
            <properties>
              <property name="location" value="${jboss.server.data.dir}${/}search"/>
            </properties>
          </loader>
          <loader class="org.infinispan.loaders.cluster.ClusterCacheLoader"/>
        </loaders>
      </default>
      <!-- *************************************** -->
      <!--  Cache to store Lucene's file metadata  -->
      <!-- *************************************** -->
      <namedCache name="LuceneIndexesMetadata">
         <clustering mode="replication">
            <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000" />
            <sync replTimeout="25000" />
         </clustering>
      </namedCache>
      <!-- **************************** -->
      <!--  Cache to store Lucene data  -->
      <!-- **************************** -->
      <namedCache name="LuceneIndexesData">
         <clustering mode="replication">
            <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000" />
            <sync replTimeout="25000" />
         </clustering>
      </namedCache>
      <!-- ***************************** -->
      <!--  Cache to store Lucene locks  -->
      <!-- ***************************** -->
      <namedCache name="LuceneIndexesLocking">
        <clustering mode="replication">
           <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000" />
           <sync replTimeout="25000" />
        </clustering>
      </namedCache>     
    </infinispan>
  </infinispan-config>  
{code}

Anyway I'll continue with testing clustering setup with jboss 6 and will tackle this issue afterward.

Thanks again.

[sannegrinovero]

some dangers I've spotted:

• Why are you using a ClusterCacheLoader ?
• If you're testing on the same machine, and using a FileCacheStore, you should not have both configuration share the same path to store the cache store.
• Values stored in a LuceneIndexesLocking Cache should not use a CacheLoader
• You should not define the CacheLoader options on the default cache, but make different options explicit on each cache: if you move the cache loader definition from the default cache to the other two caches you'll also be able to define different parameters to each of those caches, that will turn out useful when you'll do some performance tuning. The simplest you can apply is to have them store the files in different directories.

Why are you using JBoss6 ? It doesn't look like you're doing some maintenance activity on an older application; for newer applications I'd recommend to switch to JBoss7 and Infinispan 5; Hibernate Search 4 is coming very soon (end of September, maybe October) and you can already try out the first Alpha, this would be a perfect time to provide some feedback on it as I hope to make all the clustering much easier to setup.

[ekfliu]

I copied the ClusterCacheLoader from http session configuration, probably incorrect. I will remove it.

I am actually just testing one instance on one machine, so there is no second instance pointing to the same location.

Most of the config I copied from default-hibernatesearch-infinispan.xml in the hibernate search infinispan jar. In that configuration they do have replication enabled for LuceneIndexesLocking... Also I think default Loader is okay for now since this is just for testing, also the store does create three different directory for the caches so I dont believe conflict will occur there.

Thanks.

[sannegrinovero]

I am actually just testing one instance on one machine, so there is no second instance pointing to the same location.

Most of the config I copied from default-hibernatesearch-infinispan.xml in the hibernate search infinispan jar. In that configuration they do have replication enabled for LuceneIndexesLocking...

Yes I know: I wrote those files

But replication is correct, what I said above is that should remove the CacheLoader from the locking cache.

Also I think default Loader is okay for now since this is just for testing, also the store does create three different directory for the caches so I dont believe conflict will occur there.

Right, as long as you're not testing more than a single node on the same computer.

Please let me know if it works now, and which version you're testing. I'd suggest to try Hibernate Search 4.0.0.Alpha1 as it uses Infinispan 5 instead of Infinispan 4.2.

[ekfliu]

Here is the new config
{code:xml} 
<!-- hibernate search index definitions -->
  <infinispan-config name="hibernateSearch" jndi-name="java:CacheManager/searchIndex">
    <alias>hibernate-search-cache</alias>
    <infinispan xmlns="urn:infinispan:config:4.2">
      <global>
        <transport clusterName="${jboss.partition.name:DefaultPartition}-Search" distributedSyncTimeout="17500">
          <properties>
            <property name="stack" value="${jboss.default.jgroups.stack:udp}"/>
          </properties>
        </transport>
        <globalJmxStatistics enabled="true"/>
        <shutdown hookBehavior="DONT_REGISTER"/>
      </global>
      <default>
        <locking lockAcquisitionTimeout="20000" writeSkewCheck="false" concurrencyLevel="500" useLockStriping="false" />
        <jmxStatistics enabled="true"/>
        <lazyDeserialization enabled="false" />
        <invocationBatching enabled="true"/>
        <eviction maxEntries="-1" strategy="NONE" />
          <expiration maxIdle="-1" />
        <clustering mode="replication">
           <!-- Prefer loading all data at startup than later -->
           <stateRetrieval timeout="20000" logFlushTimeout="30000" fetchInMemoryState="true" alwaysProvideInMemoryState="true" /> 
           <!-- Network calls are synchronous by default -->
           <sync replTimeout="20000" />
        </clustering>        
      </default>      
      <!-- *************************************** -->
      <!--  Cache to store Lucene's file metadata  -->
      <!-- *************************************** -->
      <namedCache name="LuceneIndexesMetadata">
         <clustering mode="replication">
            <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000" />
            <sync replTimeout="25000" />
         </clustering>        
         <loaders passivation="false" shared="false" preload="true">
          <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true" purgeOnStartup="false">
            <properties>
              <property name="location" value="${jboss.server.data.dir}${/}search"/>
            </properties>
          </loader>
        </loaders>         
      </namedCache>
      <!-- **************************** -->
      <!--  Cache to store Lucene data  -->
      <!-- **************************** -->
      <namedCache name="LuceneIndexesData">
         <clustering mode="replication">
            <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000" />
            <sync replTimeout="25000" />
         </clustering>
         <loaders passivation="false" shared="false" preload="true">
          <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true" purgeOnStartup="false">
            <properties>
              <property name="location" value="${jboss.server.data.dir}${/}search"/>
            </properties>
          </loader>
        </loaders>
      </namedCache>
      <!-- ***************************** -->
      <!--  Cache to store Lucene locks  -->
      <!-- ***************************** -->
      <namedCache name="LuceneIndexesLocking">
        <clustering mode="replication">
           <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000" />
           <sync replTimeout="25000" />
        </clustering>
      </namedCache>     
    </infinispan>
  </infinispan-config> 

                                      
            
                                                                          
                                                                                    
            
                                                                                                                                                   
                                                                         
       
{code}

I did some additional test the problem is the IndexReader is never refreshed in a jms setup.

SharingBufferReaderProvider.java line 265

{code}
IndexReader beforeUpdateReader = current.reader;
                    try {
                         updatedReader = beforeUpdateReader.reopen();
                    }
{code}

This always return the same IndexReader no matter how many update to index is done.

{code}
  public boolean isCurrent() throws CorruptIndexException, IOException {
    ensureOpen();
    if (writer == null || writer.isClosed()) {
      // we loaded SegmentInfos from the directory
      return SegmentInfos.readCurrentVersion(directory) == segmentInfos.getVersion();
    } else {
      return writer.nrtIsCurrent(segmentInfosStart);
    }
  }
{code}

The SegmentInfos always return the same version. So the IndexReader is never updated. Strangely it works fine under regular lucene setup.

 

[sannegrinovero]

The SegmentInfos always return the same version. So the IndexReader is never updated. Strangely it works fine under regular lucene setup.   

Very strange, thanks for looking into that.

Could you try the not-shared IndexReader configuration too:

http://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#configuration-reader-strategy

Are you sure the nodes are connected? You should enable debug level logging on JGroups and look for cluster members on the node which is not able to find the updates; among the members you should see the master listed too.

If that is not the problem please open an issue on https://hibernate.onjira.com/browse/HSEARCH

[ekfliu]

I did some additional debugging, narrow it down to the FileListOperations.java which seems to be giving two different version of the file list depends if the ejb is calling or war application calling the method. I created two sar files each reading and writing to each other cache meta data mimicing what the FileListOperations is doing and it is working correctly each is able to see each other changes.

So I continue all the way down to the DefaultDataContainer class of infinispan, and it turn out ConcurrentMap in DefaultDataContainer was able to hold two different key with the same hashcode!!!. I checked both ejb and war application are getting the same DefaultDataContainer instance and same entries field ConcurrentMap. The FileListCacheKey of both are different instances the value of the ImortalCacheEntry are also different Instances. I verify although the key instances are different they generate the same hashcode.

I am not sure how it is possible a ConcurrentMap is able to hold two keys with the same hashcode. Could be a bug with JDK, could be class loading issues, could be concurrency issues. So basically for some reason the same DefaultDataContainer instance was able to serv both ejb and war application as if they both have their own little map.

Will upgrade jdk to latest 1.6 maybe move ejb mdb to regular sar application maybe that will fix class loading issues.

[ekfliu]

attempted update to jdk 1.6 as well as moving MDB to a regular sar application made no difference.

The bug seems to be located at DefaultDataContainer.java line 94

{code} 
   public InternalCacheEntry peek(Object key) {
      InternalCacheEntry e = entries.get(key);
      return e;
   }
{code}

InternalCacheEntry return different cache entry for the key object that generate the same hashCode. Unfornately debugging the actual java.util.ConcurrentHashMap is not possible. But I did attempt to do the following in debug Expression

key.hashCode() == new org.infinispan.lucene.FileListCacheKey("xyz").hashCode() eval to true

entries.get(new org.infinispan.lucene.FileListCacheKey("xyz")) eval to null

entries.get(key) eval to actual wrong object

I am not sure how that is even possible, but i dont think i will waste time on this anymore.

Thanks

[sannegrinovero]

I did some additional debugging, narrow it down to the FileListOperations.java which seems to be giving two different version of the file list depends if the ejb is calling or war application calling the method. I created two sar files each reading and writing to each other cache meta data mimicing what the FileListOperations is doing and it is working correctly each is able to see each other changes.

That is the interesting part. What do you mean with "different version" ? Do they contain a different list or are they just different instances?

So I continue all the way down to the DefaultDataContainer class of infinispan, and it turn out ConcurrentMap in DefaultDataContainer was able to hold two different key with the same hashcode!!!. I checked both ejb and war application are getting the same DefaultDataContainer instance and same entries field ConcurrentMap. The FileListCacheKey of both are different instances the value of the ImortalCacheEntry are also different Instances. I verify although the key instances are different they generate the same hashcode.

I am not sure how it is possible a ConcurrentMap is able to hold two keys with the same hashcode. Could be a bug with JDK, could be class loading issues, could be concurrency issues. So basically for some reason the same DefaultDataContainer instance was able to serv both ejb and war application as if they both have their own little map.

Will upgrade jdk to latest 1.6 maybe move ejb mdb to regular sar application maybe that will fix class loading issues.

The uniqueness in a Map is not mandated by the hashcode, what you see is expected. Uniqueness is mandated by the equals() implementation on the key.

Being an HashMap is just an implementation detail for a Map contract, which means it organizes the internal data in buckets according the the key's hashcode, but it's totally possible to store more than one key in the same bucket. And of course there are other Map implementation which don't care at all for the hashcode.

[ekfliu]

You're right the hashcode alone cannot determine the actual key object in a map since a collision could occur, you will still need to do equals with key itself if multiple key are in the bucket.

{code}
@Override
   public boolean equals(Object obj) {
      if (this == obj)
         return true;
      if (obj == null)
         return false;
      if (FileListCacheKey.class != obj.getClass())
         return false;
      FileListCacheKey other = (FileListCacheKey) obj;
      return indexName.equals(other.indexName);
   }
{code}

This mean its probably an issues with classloader as the indexName is the same the only way to fail the equal is if class is not equal.

[sannegrinovero]

So you're having the equals fail for the same indexName ?

Yes that's a great observation.

Do you have multiple copies of the same jar on the classpath? you should include the lucene-infinispan-directory only once with your application

[ekfliu]

Yeah indexName are the same, I am not smart enough to get two different indexName to generate the same hashCode().

I wonder if you replace

if (FileListCacheKey.class != obj.getClass())

with

if (!FileListCacheKey.class.equals(obj.getClass()))

would that make a difference as far as class loader is concerned....

[sannegrinovero]

Yeah indexName are the same, I am not smart enough to get two different indexName to generate the same hashCode().

might be bad luck

I wonder if you replace

if (FileListCacheKey.class != obj.getClass())

with

if (!FileListCacheKey.class.equals(obj.getClass()))

would that make a difference as far as class loader is concerned....

No, and it's not really the point. you must stricly not have two definitions of the same class in the same application, or you'll have many more hard to debug issues, not only with Infinispan but just any code.

Please check your jars, at least one is duplicate and should be removed.