When Node A adds an entry in to the Cache.  The Listener I have attached to the Cache, fires on Node A only.

Nothing happens on Node B.

Have you registered the cluster listener only in Node A? If so, that's how a clustered listener is supposed to work. A clustered listener get's a single invocation for each updates on either Node A or Node B.

When you register a listener that is clustered it will receive all events from any node. So in this case your listener on A and listener on B would both be notified of the event.

So I am not sure why you didn't get it on both. Is the data that was written accessible from both nodes after the put completes?

So this sounds like the cache is not clustered properly. I can't tell from what you posted here why that is. Do you have more than one cache definition?

But the reason both nodes can see it seems to be because you have a shared cache store so they can see the data the other node wrote by accessing it from the database.

If you enable TRACE level and capture the output, it should tell us pretty well.

I'll +1 the issue with clustered listeners.  I need to get a sample project together that demonstrates the problem.  I had a distributed cache that would never, ever, ever, send @CacheEntryCreated events.  In fact it would send @CacheEntryModified events in their place so I simply listened to that (in my use case I didn't need to distinguish between creates and updates so that was fine.

In another case, it would only occasionally send @CacheEntryCreated events, and would sometimes not send @CacheEntryModified events.  I removed that listener entirely and fired a JMS message over a topic to notify of cache entry puts.

This is a project several years in development now with a few dozen caches and I can't imagine a clustering issue going unnoticed by everything but these cache entry event listeners.  Hopefully I'm able to reproduce it in a sample project.

What would be helpful, and possibly faster, is if I had a set of log categories I could set to TRACE that might (hopefully) relate to events so I could do some diagnosing myself.  Any thoughts?

Scott, I have to admit I am surprised to hear that, cluster listeners has been around for over 4 years now and it is the first I have really heard of such things. The main reason I always find for events not being fired are because of the cluster not being formed as in the above post. There is one thing to note though, that a CREATE event can turn into a MODIFY if the primary owner crashes in the middle of processing a write with a non transactional cache, but that should be extremely infrequent (when this occurs the CacheEntryModifiedEvent (Infinispan JavaDoc All 9.4.0.Final API)  will be true btw).

If you are able to get something to demonstrate the issue, we can definitely try to figure out what is going on!

Huh... I'm also getting 2 POST-@CacheEntryExpired events on a perfectly healthy cluster.  Definitely going to look into this more before truly suggesting it's related. It happens on 2 different threads within the same millisecond, too.  The listener itself is declared @Listener( primaryOnly = true ).  I remember reading something about multiple expiry events in the docs but I thought that was during a failover or something.  Will keep you posted on the main issue.

Unfortunately expired events are a bit special. They are initiated outside of locking and thus multiple concurrent reads can cause more than one event. And we err on the side of giving possibly an extra one just in case. This can be especially apparent when doing a read from a non owner node. In this case this could cause a read from a primary and backup node for example. If both of these see the expired entry, they may both cause an expired event. One thing to note is that these events are ordered internally, so you don't have to worry about an expired, create and expired event, for example.

There were some more improvements in 9.4.0 as well around expiration, but unfortunately under concurrent reads for an expired read, there is a chance of getting more than one event.

It's definitely a difficult problem to solve.  Now in my case the listener isn't clustered; would these duplicates disappear if I just set clustered=true instead of primaryOnly=true? Funny I'm also seeing CacheEntryExpiredEvent.getValue() return null...

Wait a minute.... if I have primaryOnly=false and clustered=true for a listener, how could @CacheEntryExpired get fired on TWO different nodes for the exact same key (one getValue()==null).  Is that because it's a distributed cache? Wouldn't there still be exactly one primary owner that doesn't change for a given key assuming a healthy cluster?

Is there any way my cluster configuration could be messed up enough to break listener functionality but work just fine on absolutely everything else?

Edit: Looks like the null itself is expected: https://blog.infinispan.org/2015/10/expiration-enhancements.html

     "[...] the first event will contain the value (if applicable) and any others will show a null value."

silvaran  wrote:

 

Wait a minute.... if I have primaryOnly=false and clustered=true for a listener, how could @CacheEntryExpired get fired on TWO different nodes for the exact same key (one getValue()==null).  Is that because it's a distributed cache? Wouldn't there still be exactly one primary owner that doesn't change for a given key assuming a healthy cluster?

A cluster listener is special in that it receives all notifications you are registered for and that pass the provided filter if present, irrespective of what node it is registered on. If you had the event fired on two different nodes, it sounds like you have two different clustered listeners. The event received by cluster listeners will always be the same unless there is a topology change and then one or more nodes may or may not get a duplicate, with the retry flag set (not counting expiration).

silvaran  wrote:

 

Edit: Looks like the null itself is expected: https://blog.infinispan.org/2015/10/expiration-enhancements.html

     "[...] the first event will contain the value (if applicable) and any others will show a null value."

Yes, this is what I was referring to about concurrent access. Technically you can also get a null value if an entry expires from a cache store and isn't present in memory. This is a legacy thing that was done for performance a long while back, and haven't had a chance to fix it yet.