Is this question about Infinispan's tree module, or JBoss Cache?
Sorry it wasn't clear, this is with Infinispan's tree module.
the error is due to the 2nd cluster node thinking that the child is "fully" in the cache when it isn't. For instance, getChildrenNames() will return the the "/foobar" node even if if it only has "key1" (i.e. it hasn't gotten key2 or key3 yet, so the MyObject() constructor gets an error when trying to dereference one of the missing keys. I suppose the constructor could throw an exception and then the could above could step-over the entry, but I was hoping there would be a better way within treecache - and ideas?
I think that's your best option when working with the tree API. If working with lower level API (e.g. the Cache directly) you could use the locking API to prevent concurrent modifications. Actually thiking more about it, another approach would be to have a convention that, when either reading or writing a tree node, to acquire a certain lock (e.g. the FQN).
the error is due to a missing reader-writer lock - the removeNode should block until all readers are out and block new readers from getting in. If it was on the same JVM, I'd use standard synchronization mechanism, but being distributed, what is the best solution?
again the synthetic lock described above should do.