Infinispan cache is not built immediately after invalidation (wildfly 10.0.0.Final and wildfly 10.1.0.Final)

[wwang2016]

Hi,

I have identified following issue in a cluster of two instances with two users on separate instances.

(1) user 1 login to wildfly instance 1

(2) user 2 login to wildfly instance 2

The following is expected:

(3) user 1 searches for a list of objects (hotel), the cacheable query is printed in instance 1

(4) user 2 searches for a list of objects (hotel), the cacheable query is printed in instance 2 (same query)

(5) user 1 visits hotel (id = 1), no query was printed (this is through entityManager.find). This is expected since the query to search for a list of hotel should already cached the object

(6) user 2 visits hotel (id = 1), no query was printed (this is through entityManager.find). This is expected since the query to search for a list of hotel should already cached the object

(7) user 1 go back to the search page, no query is printed out although the page execute the query to get a list of hotels, this is good since the query is cached.

(8) user 2 go back to the search page, no query is printed out although the page execute the query to get a list of hotels, this is good since the query is cached.

The following is still expected for user 1:

(9) user 1 visit hotel (id=1), no query was printed out,  user 1 edited the hotel by changing the value of a property, the update statement was printed out,

(10) user 1 go back to search page, the query to search for a list of hotels was printed out. This is expected since the cache needs to be re-built

(11) if user 1 visit the hotel again, and go back to search page, no query would be printed out.

Summary: everything is good for user 1

The following is still expected for user 2:

(12) user 2 visit hotel(id=1), the query to get hotel by id is printed out, this is expected since the cache should be invalidated and instance 1 should have send a signal to instance 2 to invalidate the cache, so the query will be executed.

(13) user 2 go back to search page, the query to get a list of hotel was printed out, and this is still expected since the cache should be invalidated and the query should be executed again

Now, the unexpected:

(14) user 2 visit hotel(id = 1) again, the query to get the specific hotel by id is printed out !!!

(15) If  user 2 go back to search page, the query to get a list of hotel was not executed (Good), but the query to get the specific hotel by id is printed out again.

This behavior will continue until about three minutes (or less) later, no query will be printed out. I assume that the cache SHOULD be re-built immediately in instance 2 instead of a delay of about 3 minutes or less?

I posted a similar topic before. However, the code and logic was bit complicated, so I cleaned up code to make the use case much simpler, and I still get the same issue. This was tested in wildfly 10.0.0.Final and wildfly 10.1.0.Final.

I also tried to set default-cache of hibernate cache-container to replicated cache (in the configuration file, it is timestamps), and same issue happened

Any idea?

Thanks

[wwang2016]

I would like to add some investigation through checking the cache in jconsole

(1) after user #1 edited the object, the numberOfEntries (cluster wide) reduced by 1, indicating the other instance got the notification, and invalidated the object.

(2) user #2 visit the object many times, and could not build cache since the query got executed every time. When I checked the numberOfEntries, it stayed at less than 1

(3) afterwards, the cache expired, and user #2 can build cache again and this issue disappear

Summary, it looks like once the entity is invalidated, further query will not be able to build up the cache for this object in the instance receiving the notification to invalidate the object. However, this does not happen with the instance that actually have the object modified.

Anyone has experience this before? Is this supposed behavior?

Thanks