Locking strategy suggestion

[yelin666]

In our product, we use Fuse ESB, and its cluster for redundancy and load balancing. When an event is generated on the bus, all the subscribers across the cluster would get the event. However, we expect only one subscriber in the cluster to handle any particular event, and are planning to use the Infinispan cluster-wide lock to support it. Basically, only the instance, which acquires the cluster-wide lock, could proceed to process the event. However, when all subscribers get the event around the same time and race for the lock, they may all fail to get the lock with timeout, as each of them acquires the local lock but fails to get locks on remote instances. What strategy would be recommended for this use case, when multiple instances race for the same lock at the same time, and we expect one of them to get the lock without much delay?

[mircea.markus]

Enabling deadlock detection should decrease the chance for this to happen. Do you DLD on?

[yelin666]

Thanks for the response. I'll try DLD later, and see how it helps.

[galder.zamarreno]

Infinispan doesn't sound to be the right tool for this job. I don't see you mentioning that you store the event in Infinispan. You seem to use Infinispan simply as a distributed lock manager, in which case I think you'd be better off using JGroups' distributed lock manager. The downside though is that I don't think JGroups has DLD.

[yelin666]

Galder, thanks for sharing your thoughts. Based on our experience with the Infinispan lock so far, even before we saved the key/value pair to the cache (basically, get(key) returns null), we can still call lock(key) and that is nice. In our use case, we expect the lock to be event specific, and I doubt JGroups' distributed lock manager could support that. I assume Infinispan lock is still our best choice. Please let me know if my assumption is correct.

I got a turn around from another project to be able to test DLD now, what LockAcquisitionTimeout for lock in general and SpinDuration for DLD would you suggest to try? Response would be appreciated.

Manik, please chip in on this topic if you feel necessary.

[mircea.markus]

The downside though is that I don't think JGroups has DLD.

My impression was that Bela would implement this soon. It's much easier to implement  (compared with ISPN) as all the locks reside on the same server in the recommended jgroups lock protocol.

[mircea.markus]

I got a turn around from another project to be able to test DLD now, what LockAcquisitionTimeout for lock in general and SpinDuration for DLD would you suggest to try? Response would be appreciated.

I'm afraid there's no straight answer to this: the smaller the SpinDuration the more CPU you use for DLD. It all depends on your data access pattern, key contention, transaction size (i.e. how many keys do you have per transaction). My suggestion is "measure, don't guess": try different values and see what fits you better. If you need to run distributed benchmarks consider using Radargun: radargun.sourceforge.net

[galder.zamarreno]

...In our use case, we expect the lock to be event specific, and I doubt JGroups' distributed lock manager could support that...

I have no idea what that means...

[yelin666]

Basically, we'd like the lock to be key based, a lock for a specific key. Does that make sense?

For JGroups' distributed lock, is it a single global lock for everything?

[galder.zamarreno]

Lin, I'd recommend that you check the JGroups source code, but I seriously doubt Mr Ban has implemented Distributed Locking as  a single global lock.

In fact, as you can see in https://github.com/belaban/JGroups/blob/master/src/org/jgroups/blocks/DistributedLockManager.java#L224 , whenever you lock you give a lock id, so the lock is named, rather than being global.

[yelin666]

I did check out the JGroups distributed locking, and realized it's id based.

My question is what's the relationship between Infinispan lock and the JGroups distributed lock. We are using Infinispan for data storage anyway, so what's the cons and pros of using each of them?

[galder.zamarreno]

There's no relationship between them. My concer was that you're using Infinispan just as a way to do cluster wide locking, without the need of doing data storage. If you need data storage, then your use case is fine, but this was not obvious from the start. Hence why I suggested JGroups if all you want to do is cluster wide locking.