2 Replies Latest reply on Mar 13, 2014 1:34 PM by jugglingcats

AdvancedCacheLoader.process() method -- purpose and issue for very large store

jugglingcats Mar 13, 2014 10:37 AM

Hi, I have an implementation of ExternalStore for Mongo that does some object mapping specific to our requirements.

When I started using async mode in our configuration, Infinispan told me the store had to implement AdvancedCacheLoader... so I implemented process() and size() methods no problem based on the SingleFileStore example, but was a bit confused why/when they would be invoked.

Now we're performance testing and one of our caches has 200,000,000 entries or so in Mongo, and the process() method is being called for this cache on start / on joining the cluster. Because we have to interate all entries, this is taking a loooonnng time! In all cases the fetchValue and fetchMetadata parameters are false, so am not sure exactly what it's doing and whether it's necessary.

The large cache in question is in DIST_ASYNC mode and most of the caches use this mode with async enabled. We do have one cache that is REPL_SYNC using the same store implementation with async disabled.

As a workaround I ignore the process call if both fetchValue and fetchMetadata are false, but not sure if this is valid thing to do!! It would be great to get a fuller explanation of the purpose of this method.

Many thanks, Alfie.

1. Re: AdvancedCacheLoader.process() method -- purpose and issue for very large store

vblagojevic Mar 13, 2014 12:35 PM (in response to jugglingcats)

Hi,

What seems to be happening is that you are warming up cache on startup. Is that what you really want to do? Have a look at http://infinispan.org/docs/6.0.x/user_guide/user_guide.html#_configuration_2 to understand all the options in detail.

If that does not help please do say so and we'll have another look.

Regards,
Vladimir
Actions
2. Re: AdvancedCacheLoader.process() method -- purpose and issue for very large store

jugglingcats Mar 13, 2014 1:34 PM (in response to vblagojevic)

Thanks for the reply. I am not specifying preload() and the default is false, so am not consciously trying to warm up the cache. I do have fetchPersistentState(true) in my configuration, but seem to remember without this I was not getting any loading from store happening at all (it was a while ago). The javadoc says "Finally, setting it to true only makes sense if in a clustered environment, and only 'replication' and 'invalidation' cluster modes are supported.", and my large cache is DIST_ASYNC. I am not specifying shared(true) which I probably should be since Mongo is shared like JDBC -- not sure if this is significant though.

It would be good to get an explanation of when/why the process method is called on AdvancedCacheLoader. The documentation doesn't make it 100% clear.

Your help much appreciated.
Alfie.
Actions

Go to original post