Recommendations on usage of invalidation cache

[stianst]

In Keycloak we use invalidation caches heavily to prevent database traffic. However, we have two related problems that should be a common issue with invalidation caches and I'm looking for some advice and best practices on how to deal with these:

Problem A - Node B loads entity from DB imediately after Node A updates/invalidates the entity

T1 Node A loads entity from db and adds to cache

T2 Node B loads entity from db

T2 Node A updates entity and invalidation message is sent to cluster

T3 Node B adds entity to cache

In this case in T2 Node B has loaded the stale data from the database, but misses the invalidation message as it's not been added to the cache yet.

Problem B - Node A and Node B changes entity at the same time (we're not bothered about one node overwriting changes made by the other, but we need both nodes and database to be consistent)

T1 Node A loads entity from db and adds to cache

T1 Node B loads entity from db and adds to cache

T2 Node A updates entity and invalidation message is sent to cluster

T2 Node B updates entity and invalidation message is sent to cluster

T3 Node A adds update entity to cache

T3 Node B adds update entity to cache

In this case in T3 Node A and B both adds the value they believe the entity has, but only one matches the db.

[rvansa]

Hibernate ORM Second-level cache implementation deals with the same problem. For invalidation mode cache, it uses side-cache to track about-to-happen DB reads and invalidations, and introduces a cluster-wide lock-unlock phase around DB write. The cache updates check this info and drop the update if there is a concurrent modification (so the threads that do DB read are never blocked, just skip cache update). The code is about here [1][2] and took quite some time to get right.

There are other approaches for replicated (or distributed) caches, described a bit in this blogpost [3] - using expiring objects to represent a missing entry, and sometimes taking advantage of versions to remove the need for the two-phase (at the risk of stale reads).

I am sorry that there's no OOTB solution, but maybe the ORM 2LC code could be refactored in order to make it usable by both 2LC and Keycloak.

[1] hibernate-orm/PutFromLoadValidator.java at master · hibernate/hibernate-orm · GitHub

[2] hibernate-orm/InvalidationCacheAccessDelegate.java at master · hibernate/hibernate-orm · GitHub + inheriting classes

[3] Infinispan: Hibernate Second Level Cache improvements