In JBossCache 3.2.1 I have a clustered cache configuration based on mvcc-entity but I added a cache loader. I am trying to use this to replace an old page content cache system. As a web page's content is collected the results are cached. The next visit will load the content from page and render the page. Pretty straight forward so far. The cache is arranged in a hierarchy, for this example lets say /SITE/AREA/ID. All content is cached at leaf node in the cache under the ID. If someone updates some content on the site we are calling removeNode("/SITE"). This replicates and invalidate node command to the other nodes in the cluster. Without a cache loader this all works great. The other clusters invalidate the node and all its children. The problem comes up when in introduce a cache loader. We want the cache loader because the cached content can be multiple GBs in a very short period of time. We want only the active stuff in the memory cache and the less active stuff evicted to the store. This all behaves fine until we delete the /SITE node.
When the /SITE node is deleted on host A it wipes it from memory and from the loader. It then send the invalidate to host B. Host B marks the node and its children as invalid in memory but does nothing with the cache loader and the content remains in the loader. If host B is requested for content under /SITE it sees that the in memory /SITE is invalid and loads it from the cache loader. At this point it loads the old stale data out of the loader since it never gets wiped.
To make matters worse it appears that the in memory cache of /SITE on B is fubared in that it will never invalidate locally after this point. If the removeNode is called on host B it fails silently leaving the content cached in /SITE.
I have tried both shared and non-shared cacheloaders with the same result. Removing the cacheloaders works but is not a complete solution for our needs.
If I make every node a leaf node it works fine, but then we can't just whack all cache for /SITE and its children which makes cache invalidation impossible given the already sketchy nature of the cache I am replacing.
Has anyone successfully use invalidation with a cacheloader? It seems like an obvious combination especially when the cacheloader is shared.