1 Reply Latest reply on Feb 9, 2013 7:29 PM by pjakub

    Modeshape 3.1.1.Final Clustering Index problem

    pjakub

      Hi,

      I'm using latest modeshape 3.1.1.Final + Infisnispan 5.1.2.Final + HibernateSearch 4.1.1 and I'm trying to get infinsispan indexing to work.

       

      My configuration is as:

      jcr-config.json:

      {

          "name" : "tlk_repo",

          "jndiName" : "",

          "storage" : {

              "cacheName" : "tlk_infinispan_repo",

              "cacheConfiguration" : "infinispan_configuration.xml",

              "binaryStorage" : {

               "type" : "cache",

                  "dataCacheName" : "dataCache",

                  "metadataCacheName" : "metadataCache"

              }

          },

          "query" : {

              "enabled" : true,

              "enableFullTextSearch" : true,

              "indexStorage" : {

                  "type" : "infinispan",

              "cacheConfiguration" : "/path_to_my/infinispan_configuration.xml"

              },

              "indexing" : {

                "mode" : "sync",

              "rebuildOnStartup" : { "when":"if_missing"}

             

              },

          },

          "workspaces" : {

              "predefined" : ["default","global"],

              "default" : "default",

              "allowCreation" : true

           },

          "security" : {

              "anonymous" : {

                  "roles" : ["readonly", "connect","admin","readwrite"],

                  "useOnFailedLogin" : false

              }

          },   

       

          "node-types" : ["tlk-ejb.jar/tlk-custom-node-types.cnd"],

       

          "clustering" : {

              "clusterName" : "modeshape3",

          "channelConfiguration" : "ms-jgroups.xml"

          }

      }

       

      and my infinispan configuration is:

       

      <?xml version="1.0" encoding="UTF-8"?>
      <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"
          xmlns="urn:infinispan:config:5.1">
      
          <global>
              <transport clusterName="modeshape-grid">
                  <properties>
                      <property name="configurationFile" value="path_to/jgroups-modeshape.xml" />
                  </properties>
              </transport>
          </global>
      
          <default>
              <clustering mode="distribution">
                  <sync />
                  <hash numOwners="2" rehashWait="120000" rehashRpcTimeout="600000" />
                  <l1 enabled="false" lifespan="600000" />
              </clustering>
      
              <!-- Invocation batching is required for use with the Lucene Directory -->
              <invocationBatching enabled="true" />
      
              <transaction
                  transactionManagerLookupClass="org.infinispan.transaction.lookup.DummyTransactionManagerLookup"/>
              <loaders passivation="false" shared="true" preload="false">
                  <loader class="org.infinispan.loaders.remote.RemoteCacheStore" fetchPersistentState="false"
                         ignoreModifications="false" purgeOnStartup="false" >
                        <properties>
                       <property name="hotRodClientPropertiesFile" value="hotrod-client.properties"/>
                       <property name="remoteCacheName" value="tlk_content"/>
                        </properties>
                   </loader>
              </loaders>
          </default>
      
          <namedCache name="tlk_repo-index-lock">
              
          </namedCache>
      
      </infinispan>
      

       

       

      and I'm getting following error:

       

      17:52:48,761 ERROR [LogErrorHandler] HSEARCH000058: Exception occurred org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@559e4f

      Primary Failure:

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image  Work Type  org.hibernate.search.backend.UpdateLuceneWork

      Subsequent failures:

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/height/Long/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/parentLovId/Long/*  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/userId/Long/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/description/String/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/keywords/String/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/width/Long/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/name/String/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/smallThumbnail/Reference/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}thumbnail  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/departmentID/Long/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}thumbnail/height/Long/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/updateDate/Date/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/archived/Boolean/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/updateUserId/Long/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/global/Boolean/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}thumbnail/width/Long/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}thumbnail/originalImage/Reference/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7jcr:nodeTypes  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/bigThumbnail/Reference/1  Work Type  org.hibernate.search.backend.UpdateLuceneWork

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7/jcr:system/jcr:nodeTypes/{httpmynodes}image/lovId/Long/*  Work Type  org.hibernate.search.backend.UpdateLuceneWork

       

      org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@559e4f

          at org.apache.lucene.store.Lock.obtain(Lock.java:84)

          at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108)

          at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:127)

          at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:102)

          at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:119)

          at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:99)

          at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67)

          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

          at java.util.concurrent.FutureTask.run(FutureTask.java:138)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:662)

      17:52:48,820 ERROR [LuceneBackendQueueTask] HSEARCH000072: Couldn't open the IndexWriter because of previous error: operation skipped, index ouf of sync!

      17:52:49,900 ERROR [LogErrorHandler] HSEARCH000058: Exception occurred org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@559e4f

      Primary Failure:

          Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id de0a805317f1e7mode:namespaces  Work Type  org.hibernate.search.backend.UpdateLuceneWork

       

      org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@559e4f

          at org.apache.lucene.store.Lock.obtain(Lock.java:84)

          at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1108)

          at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:127)

          at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:102)

          at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:119)

          at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:99)

          at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67)

          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

          at java.util.concurrent.FutureTask.run(FutureTask.java:138)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:662)

      17:52:49,907 ERROR [LuceneBackendQueueTask] HSEARCH000072: Couldn't open the IndexWriter because of previous error: operation skipped, index ouf of sync!

       

      When I'm running one server instance, everything works fine, when I'm starting second, I'm getting that exception. Is there some infinispan option, that I'm missing, or is it some bug ?

      If anyone managed to run it in that configuration I'd be glad to see sample working solution.

        • 1. Re: Modeshape 3.1.1.Final Clustering Index problem
          pjakub

          Hah!

           

          I've made it! There were some problems, I'll probably report few bugs, but it's working!

           

           

          ***********How I think that should be: ****************************

           

          when I use Infinispan for index storage, information about changes should propagate through infinispan.

          At least that's what I started to think after reading description of 'lucene' backend at schema at

           

          https://raw.github.com/ModeShape/modeshape/master/modeshape-jcr/src/main/resources/org/modeshape/jcr/repository-config-schema.json

           

          that says, that "It can also work for clustered repositories, but requires an index storage type that supports locking."

           

          If anyone know how to configure infinispan caches to make it work, I'll be very happy to hear about that.

           

          ***********How it is: ********************************************

           

          When I user Infinispan for index storage, I HAVE to configure 'query.indexing.backend' (I use 'jgroups-master',and 'jgroups-slave'. If I leave default value : 'lucene', two modeshape instances owervrites those values causing exceptions:

          HSEARCH000058: Exception occurred java.io.IOException: Read past EOF

           

          It might be, that I'm missing some option for tlk-index-locks infinispan cache, and that causes my indexes to get overwritten, but I have no idea how to make it work, through changing infinispan configuration.

           

           

          So I used 'jgroups-master' for 'query.indexing.backend'... and there were some problems too:

          I use hibernate-search version 4.1.1.Final, and it's very sensitive about it's configuration,

          and I had to change lines 211 and 213 at file

           

          org/modeshape/jcr/query/lucene/basic/BasicLuceneConfiguration.java

           

              setProperty("hibernate.search.default.worker.jgroups.clusterName", channel);

          that should be:

              setProperty("hibernate.search.services.jgroups.clusterName", channel);

           

          and:

                setProperty("hibernate.search.default.worker.backend.jgroups.configurationFile", config);

          should became:

               setProperty("hibernate.search.services.jgroups.configurationFile", config);

           

          otherwise hibernate-search would throw ugly exception and won't start...

           

           

          and that was easy part...

           

          this schema:

          https://raw.github.com/ModeShape/modeshape/master/modeshape-jcr/src/main/resources/org/modeshape/jcr/repository-config-schema.json

          is buggy. there is a big mistake for jgroups-slave, it's description is copied from jms-slave but content is not changed, and that causes that you can't configure jgroups-slave without ugly hack like that:

           

          "indexing":{

                           "hibernate.search.services.jgroups.clusterName":"backendChannel",

                          "hibernate.search.services.jgroups.configurationFile": "backend-jgroups.xml",

                          "backend":{

                              "type":"jgroups-slave",

                              "queueJndiName":"dummy",

                              "connectionFactoryJndiName":"dummy"

                          }

                  }

           

          I had to set queueJndiName and connectionFactoryJndiName because schema requires me to do so, and really important properties I had to set in hibernate.search.* properties

           

          could those things be fixed ? Or am I just using that in wrong way ?