-
1. Re: Help with ISPN + apiman
sannegrinovero Sep 1, 2015 7:36 AM (in response to eric.wittmann)Hi,
I'll let the Infinispan core team answer on your main question as I don't know why it would return a stale value, but when looking at the InfinispanRateLimiterComponent I noticed a design issue: the javadoc seem to imply that this is meant to work in a multiple-nodes cluster configuration too, but you're doing a local only synchronization.
Also you seem to be storing a mutable object in the Cache: keep in mind that the instance returned by Cache#get will be the same instance across multiple threads potentially reading from the cache, and another thread might need to transmit a copy of the instance "as is" to another node before your current thread gets to the explicit Cache#put operation. You cant' rely on your synchronization to protect you from such a race condition as there are other management threads internal to Infinispan which work on the entries, for example moving entries to different stores or nodes to balance resource usage.
You probably want to rethink this to use Cache#putIfAbsent initially rather than using the mutex, and refactor to use immutable entries rather than mutating operations such as bucket.resetIfNecessary.
-
2. Re: Help with ISPN + apiman
eric.wittmann Sep 1, 2015 9:06 AM (in response to sannegrinovero)Thanks for the feedback!
-
3. Re: Help with ISPN + apiman
william.burns Sep 1, 2015 10:07 AM (in response to eric.wittmann)Eric,
I am not sure what is going on here. Unfortunately there isn't a whole lot to tell from the trace. Is it possible you could run this printing out the arguments to the accept method as the first line of the method? I have never heard of ISPN cache returning an older value like this (except with concurrent operations), especially from a LOCAL cache which is substantially simpler than a distributed one.
Also if you are able enable tracing for ISPN should help shed some light as well.
-
4. Re: Help with ISPN + apiman
eric.wittmann Sep 4, 2015 12:14 PM (in response to william.burns)Of course - I can work on building a simple reproducer, although I'm not sure how much luck I'll have (this problem doesn't always happen and is hard to reproduce).
At the same time, can you point me at any ISPN documentation/sample that might exist which implements a simple cluster-wide shared counter?
-
5. Re: Help with ISPN + apiman
sannegrinovero Sep 4, 2015 12:42 PM (in response to eric.wittmann)Implementing a Counter is actually quite tricky on top of Infinispan's core model, and requires to think carefully about which requirements your counter has. There are plans to expose such a feature in some "easy way" on the Infinispan public API, but it's generally unclear what people mean exactly by "counter", and which trade-offs are not acceptable.
As an example, here I implemented reference counting for a shared resource lock, storing an Integer and incrementing/decrementing it as needed with atomic operations:
As far as I'm understanding you want to implement a rate limiter? That would probably need a different approach, for example you probably want a statistical estimate and not a fully accurate value?
To have an always-synchronized fully accurate value you would be generating quite some network traffic, so a better approach would be to have each node reserve some increment blocks depending on load.
To implement such a model you probably want to go at the lower level, using JGroups's COUNTER protocol:
You can either get a reference to the JGroups channel being used by Infinispan, or you can open a dedicated one, or you can multiplex a channel using the FORK protocol. If you're running this on WildFly 10, the appserver is ready to manage FORK channels so that should be the simplest approach, as you can then add a dedicated COUNTER for your application, isolated from other apps using the same network transport.
-
6. Re: Help with ISPN + apiman
william.burns Sep 4, 2015 12:44 PM (in response to eric.wittmann)Eric Wittmann wrote:
Of course - I can work on building a simple reproducer, although I'm not sure how much luck I'll have (this problem doesn't always happen and is hard to reproduce).
Sounds great, hopefully you can get lucky
Eric Wittmann wrote:
At the same time, can you point me at any ISPN documentation/sample that might exist which implements a simple cluster-wide shared counter?
Unfortunately there really isn't anything that I am aware of.
There are 2 main ways to do this.
- Pessimistic transactional cache which does a read lock - Pessimisstic.java · GitHub
- Non transactional cache using concurrent write operations Optimistic.java · GitHub
I tried to keep both examples very bare bones, but it should get the point across.
The non transactional will be faster if there is low contention for the given key or if you are invoking it on the primary owner for that key (since it only goes remote after it acquires the lock locally).
-
7. Re: Help with ISPN + apiman
eric.wittmann Sep 4, 2015 1:32 PM (in response to eric.wittmann)Great! Thanks to both of you for the responses. I'll be circling back around on this issue probably in a few weeks, at which point we'll have some time to hopefully try out these approaches.
Thanks very much for the help - I should have solicited it months ago when the implementation was first done.