Hibernate ORM Second-level cache implementation deals with the same problem. For invalidation mode cache, it uses side-cache to track about-to-happen DB reads and invalidations, and introduces a cluster-wide lock-unlock phase around DB write. The cache updates check this info and drop the update if there is a concurrent modification (so the threads that do DB read are never blocked, just skip cache update). The code is about here  and took quite some time to get right.
There are other approaches for replicated (or distributed) caches, described a bit in this blogpost  - using expiring objects to represent a missing entry, and sometimes taking advantage of versions to remove the need for the two-phase (at the risk of stale reads).
I am sorry that there's no OOTB solution, but maybe the ORM 2LC code could be refactored in order to make it usable by both 2LC and Keycloak.
 hibernate-orm/InvalidationCacheAccessDelegate.java at master · hibernate/hibernate-orm · GitHub + inheriting classes