The solution when a new member of a cluster requests and gets the entire state of the cache transferred to him obviously will not work for any decent size dataset.
What would be needed is the ability for a new member to pass a parameter to coordinator that could be used by his CacheLoader to retrieve and send only portion of the persistent state. Basically, this parameter would be a change number of some sort (or even a date) so a new member can say "ok, I already have the persistent state up to this moment, give me a delta to bring me up to speed". The true nature of this parameter can be left to the CacheLoader implementations.
After quick look at the code, it seems that such modification would require modifying underlying jgroups code.
Does this make sense at all? Is there any work being done in this direction? Would a contribution like this be welcomed?
First: we have streamin state transfer on the roadmap for 1.4 (http://jira.jboss.com/jira/browse/JBCACHE). This will break large states into smaller sized chunks.
Second: partial state transfer is also planned. If you want to send me a patch/suggestion for JGroups (Channel, JChannel, STATE_TRANSFER), go ahead and I'll see whether I can integrate it into the codebase.