What would be wrong with setting an eviction policy on the cluster so that entries are evicted older than 5 seconds? That would probably be the easiest to configure.
Though, your delay approach would probably work. Just be sure to have the removal action queued and take place in a separate thread. Some sort of batched remove might work faster.
Thank you for your answer, genman.
Evicting every 5 seconds would kill our performance, as we would rarely have a cache hit.
I've managed to get my FIXED_DELAY protocol to work. It sits on the very top of the stack and delays all Messages with getBuffer()!=null. Unit tests look good so far, I guess we'll try it this way.
Your protocol fix sounds like a pretty good solution, although I would be worried if certain messages (not related to invalidation) were held up.
Could you submit your FIXED_DELAY protocol as patch? https://jira.jboss.org/jira/browse/JGRP It's not required but may be good to have your code examined by the experts.
How does your implementation know which messages to delay? Or do you simply delay all messages? You'd want to be careful here since the only way I can think of to tell which messages to delay on a JGroups level is to deserialize everything and then look at the method ids. And delaying everything is not good since join/leave messages could be delayed as well.
Your best bet may be to write an Interceptor and add this to the start of the chain. You could create a Queue to hold all InvalidateCommands (2.2.0 and above) or MethodCalls containing the invalidate methodId (pre-2.2.0), add them to a DelayQueue and don't pass the call up the interceptor stack any further.
You could then have another Thread that polls the delay queue grab stuff off the queue as they mature, and then pass them up the interceptor stack.
I also thought that GMS messages shouldn't be delayed. My assumption from browsing through the source and debugging was that JGroups messages don't use the Message's buffer, but JBC messages do. So my condition for delaying is
if (msg.getBuffer()!=null && msg.getBuffer().length>0)
Would that be safe? If it is not, I'll have to use the Interceptor as you proposed.
It *may* be safe based on your configuration, but either way it would certainly be brittle. E.g., if you use buddy replication, JBC uses the same RPC mechanism to assign buddies, etc. And there is no good reason to delay these, and your approach won't be able to tell the difference.
The interceptor chain is the proper extension point for such things since you have a definitive idea of the command/methodcall being invoked (including payload, etc).