Workaround for eviction issue

[dev_java]

Dear Modeshape Team,

can you please recommend a workaround for the eviction issue https://issues.jboss.org/browse/MODE-2280? Do you know the reason why the infinispan team did only fix the related issue https://issues.jboss.org/browse/ISPN-4810 in the 5.x and 7.x branch but not in the 6.x where the problem was initially reported? Without using eviction there is no way to limit the memory size of persistent caches in production or did we overlook something? We tried to use infinispan 7.x and 7.1x with modeshape 4.2 but the API did change too much and it did not work out.

Thanks in advance,

Bernhard

[hchiorean]

Unfortunately there is no 100% safe technical workaround for this issue (as for why the ISPN team did not backport the fix into the 6.x community version(s) you'll have to ask this in the ISPN forums). We're also aware that eviction is really a mandatory option, unless an environment has "unlimited" RAM, which is unlikely. The only thing I can think of (since this is a concurrency issue) is to "throttle" the maximum number of concurrent interactions with the repository in your application's code - may sound easier than it actually is to do.

Because Infinispan 7 has (again) changed parts of its API from 6, ModeShape will probably move to Infinispan 7 once there a final version of Wildfly 9 (which should be using Infinispan 7). Because of the API differences the change will be significant, because ModeShape can't work at the same time with 2 different major versions of Infinispan.

[dev_java]

Hello Horia,

thank your for the quick reply. That is really bad news, it actually means that modeshape 4 cannot be used in production with a cache store?

I have not understood the complete nature of the problem, is the 'throtteling' somehow linked to the parameters that one can set for the eviction configuration (maxEntries, strategy, threadPolicy)?

Would be great if you could elaborate a bit more on the problem!

Thanks,

Bernhard

[hchiorean]

The problem surfaces only when multiple (enough) threads read & write under the same node in the repository concurrently. For example if you have a node named "parent" under which you have a large number of children and there's a lot of concurrency under the same "parent" node you may stumble across the issue (for example 100 threads read the "parent" node and at the same time another thread adds children under the parent node and this node is evicted at some point).

By "throttling" I simply meant trying to make sure the above use case doesn't happen: for example you can try to design your node structure in such a way that there are multiple "parent" nodes and you don't have a lot of concurrent access under the "same" parent (see

https://modeshape.wordpress.com/2014/08/14/improving-performance-with-large-numbers-of-child-nodes). There isn't a "magic" configuration option which will solve this problem. However, using a large-enough (depends on your use case) value for the maxEntries eviction attribute should also help a lot in alleviating this problem, as evicition will happen less often and therefore also the problem.

So https://issues.jboss.org/browse/MODE-2280  is only relevant in highly concurrent cases which operate in the same parts (nodes) of a hierarchy and nodes from this hierarchy are evicted from memory.

This can happen in a real system, but node hierarchy design together with a large-enough maxEntries value is the best (and only IMO) workaround.

[javagirlie]

Hello Horia,

thank you for the further information! I have posted a question concerning the backport in the infinispan area (https://developer.jboss.org/message/924236).
We have also put the binaryStore into infinispan, can that problem occur there too? Our "parents" that hold a larger amount of nodes are all string-based (containers nodes), these are stored in the infinispan stringStore I assume? Or can we safely enable eviction for the binaryStore and leave the stringStore as it is?

Thanks,

Susanne & Bernhard
 
Off topic: The ability to post only once in 3600 seconds here in the forums makes a discussion pretty difficult. Can that limit be lowered to say ... 10 minutes - that would be spam protection enough I think? 

[dev_java]

Hello Horia,

we have received an answer from the Infinispan guys. They promised to contact you on that specific issue:

Eviction issue with infinispan 6.x

Best regards,

Bernhard

[hchiorean]

I'll talk with the Modeshape maintainers to see whether the Infinispan version they use can be upgraded.

As mentioned in my previous post, upgrading our Infinispan version is unlikely atm. Also, we can only use ISPN >= 7.2 which hasn't even been released yet.

We have also put the binaryStore into infinispan, can that problem occur there too? Our "parents" that hold a larger amount of nodes are all string-based (containers nodes), these are stored in the infinispan stringStore I assume? Or can we safely enable eviction for the binaryStore and leave the stringStore as it is?

The problem can occur *anywhere* where data is stored in Infinispan caches with eviction enabled.

Off topic: The ability to post only once in 3600 seconds here in the forums makes a discussion pretty difficult. Can that limit be lowered to say ... 10 minutes - that would be spam protection enough I think?

This is not a setting in the ModeShape forums, I don't know where this is coming from.

[hchiorean]

In light of this discussion and the fact that the limitation of moving to a newer version of Infinispan comes from our integration with JBoss AS, we've decided to move away from this for 4.3. It means that starting with 4.3 users will have to change their Wildfly configurations a bit, but ModeShape will be able to pick up whatever the latest stable version of Infinispan is, *regardless* of the WF version.

See [MODE-2454] Change ModeShape's AS kit to start using Infinispan as a regular library not as a subsystem - JBoss Issue Tr… for more information.

[dev_java]

Hello Horia,

I think that is definitely a step in the right direction and will make the product more flexible for the community as a whole.

Thank you for the quick decision and the informative discussion.

Best regards,

Bernhard