This is not the expected behaviour; after reading the entity from DB it should get cached. In fact the invalidation in 2LC is two-phase (as opposed to vanilla Infinispan invalidation mode) to prevent you from getting stale results, so this sounds like the 'unlock' part did not finish correctly. There are also some additional checks that depend on wall-clock time, though; could you check that the wall-clock time on those machines is in sync?
The default WildFly configuration uses transactional caches wher Infinispan registers as JTA synchronzation (that's the NON_XA). 2LC now supports no-transactional caches, too, so you can try switching that to NONE.
I started the two instances in the same machine (with different sets of port number), so the time is sync.
I indeed change the out-of-box configuration from NON_XA to NONE, and I saw the instance that receives the signal to invalidate the cache was able to retrieve the latest data, and cached it since it did not print out the query twice.
Just out of curiosity, why did the setting of NON_XA broke the proper cache management?
No idea what's wrong in your case. There may be a bug that's avoided in non-transactional scenario, though the default is tested in the same way.
This may be related to the following issue I posted a while ago:
The way we worked around it is to always do a get(key) before a write operation (with a cache store enabled).
Did you describe a scenario of working on an invalidation cache directly? I was describing the default configuration of entity cache being invalidation-cache, and the instance that receives a signal to invalidate the cache (e.g. update an object in another instance) did invalidate the cache, but did not build the cache again with the latest data.
Setting NON_XA (out-of-box configuration in wildfly 10.1.0.Final) to NONE fixed the issue.
Yes, in our app we deal with Infinispan and its caches directly, and certainly with no XA. Just wanted to mention that the Cache methods in invalidation mode can return incorrect results, which could be a cause.