1 2 Previous Next 15 Replies Latest reply on Oct 12, 2010 1:39 PM by galder.zamarreno

    A question on using RAMDirectoryProvider for index

    yelin666

      We got a cache cluster in replicated mode with fetchInMemoryState set to true, and want to use RAMDirectoryProvider with indexLocalOnly set to false. When we have all cache instances up running, the data being put into the cache are indexed correctly. However, if we have a new cache instance join the cluster, the states fetching happens at the time calling cacheManager.getCache(); then when we construct a QueryHelper with the cache object passed in to add a QueryInterceptor, it will only index the objects put into the cache afterwards. Those data fetched from peers during startup won't be indexed and are not searchable. Is there a way to get those existing data before joining time indexed?

       

      Alternatively, if we don't use fetchInMemoryState, but ClusterCacheLoader for lazy loading. After we start a new cache instance, create the QueryHelper, without getting data to the local cache instance, I don't see how the exiting data would be indexed either. Please suggest how to get the exiting data indexed when using ClusterCacheLoader as well.

        • 1. Re: A question on using RAMDirectoryProvider for index
          manik

          Hi Lin

           

          re: State transfer:

           

          This is because the indexing interceptor is added to the chain after the cache starts and after the state has been transferred.  This will be properly applied after the query API is finalised, in ISPN-194.

           

          re: the clustered cache loader, you should create a JIRA for this.  If you have a unit test that would be great.

           

          Cheers

          Manik

          • 2. Re: A question on using RAMDirectoryProvider for index
            yelin666

            I am going to try the InfinispanDirectoryProvider, but not sure how to pass the index cache to the provider. Is there a good document on how to use InfinispanDirectoryProvider? So I don't have to dive into the code.

            • 3. Re: A question on using RAMDirectoryProvider for index
              manik

              I presume you've read http://community.jboss.org/wiki/InfinispanasaDirectoryforLucene

               

              Have you also seen the directory provider demo that ships with the -all.ZIP distribution?

              • 4. Re: A question on using RAMDirectoryProvider for index
                yelin666

                My questions is how to hook up the InfinispanDirectory to the QueryHelper, so I can use InfinispanDirectory for the index of my cache. Based on the parameters the QueryHelper constructor takes, it seems I can only use the Properties for the hook. I can set the "hibernate.search.default.directory_provider" to "org.infinispan.lucene.InfinispanDirectory", but it's not obvious to me what property to use to pass in the cache I want to use for the index to the InfinispanDirectory.

                 

                I hope this clarifies the confustion, and would appreciate your quick response.

                • 5. Re: A question on using RAMDirectoryProvider for index
                  sannegrinovero

                  Hello Lin,

                  the classname which you set to hibernate.search.default.directory_provider has to implement the org.hibernate.search.store.DirectoryProvider interface; you should write an implementation of that and I'm sure it will be more clear: you don't point it to the InfinispanDirectory but make a factory for it.

                  In this factory you have total flexibility in configuration: you can start a new cache, or grab one by jndi; the interface of the DirectoryProvider has an initialize method which receives the Query configuration properties too, you might use them to configure the cache, or pass the name of the cache you use in the configuration file of Infinispan.

                   

                  Sorry for this requiring to write some code, we will provide a general purpose implementation in the future but that will require some thinking and feedback to make sure it's flexible enough. Patches welcome!

                  • 6. Re: A question on using RAMDirectoryProvider for index
                    sellhorn

                    Sanne;

                     

                    We got it working but the implementation was a bit clumsy.

                     

                    We created an infinispan directory provider and in the initialize method, created an InfinispanDirectory. Problem is, I need a cache to create that directory. The cache object I wanted to use is created by the caller, so I had to pass the cache in a ThreadLocal variable to it because I can't put objects in the Properties object of initialize (it only accepts strings).

                     

                    Be happy to contribute what we did, but there's probably a much better way to do this. The solution with the threadlocal is a bit of a hack ;-)

                    • 7. Re: A question on using RAMDirectoryProvider for index
                      sannegrinovero

                      right I was wondering myself what would the best way to pass the cache; I've seen some early adopters just create the cachemanager in the initialize, this works fine if you have only one Infinispan directory but it not great if you have several.

                       

                      Another approach would be to expect the Infinispan cachemanager to be registered in JNDI as a severvice and look it up.

                       

                      You say you created the cache in the caller, so where is it initialized?

                      I guess the code for this is quite simple but could need much feedback, we need to hear about more use cases; you could bring this up on the developer mailing list of Infinispan or Hibernate Search, that's also the best place to propose patches.

                      • 8. Re: A question on using RAMDirectoryProvider for index
                        sellhorn

                        Right now I pass a cache, I probably could just get the cache from the cache manager instead. But, I create the cache manager (or managers could be multiple) centrally in an OSGi bundle, so I'm probably going to end up passing a CacheManager instance (have to do this thread local hack). Since I'm in OSGi, one approach would be to get my bundle that creates these managers, but I have no BundleContext and can't pass it to the provider. Basically, since the provider is not created by me, it's pretty hard to pass it references to anything.

                         

                        It works now, but I'm afraid what we have is not a good general solution :-(

                        • 9. Re: A question on using RAMDirectoryProvider for index
                          sannegrinovero

                          hi, did you consider or try registering the CacheManager in JNDI?

                          • 10. Re: A question on using RAMDirectoryProvider for index
                            sellhorn

                            I was hoping to avoid JNDI since I'm not running in an EE container but in OSGi (and want to support regular Java SE runtimes).

                            • 11. Re: A question on using RAMDirectoryProvider for index
                              sannegrinovero

                              right we must find a solution which is non-EE friendly too; I guess we need to elaborate a proposal to change the interface's contract, but avoiding to create a strong coupling to Infinispan.

                              • 12. Re: A question on using RAMDirectoryProvider for index
                                sellhorn

                                It would be nice if in the properties for QueryHelper;

                                 

                                queryHelper = new QueryHelper(cache, p, classesToBeIndexed);

                                 

                                That in 'p' which is a regular Java Properties object I could put an object as a property. Properties is just a hashtable, so i can do object puts. Still not super clean, but better than thread local.

                                 

                                The problem is that the hibernate code grabs that property object and converts it to some type of filtered properties, which throws invalid operations if you try to use the hashtable methods :-(. So it ensures you can only pass objects around.

                                • 13. Re: A question on using RAMDirectoryProvider for index
                                  galder.zamarreno
                                  • 14. Re: A question on using RAMDirectoryProvider for index
                                    sellhorn

                                    Galder;

                                     

                                    Thanks. Just seems a bit more complicated than just passing an instance of the CacheManager via the property object we already use though :-(

                                    1 2 Previous Next