Hello guys, i'm trying to implement a message bus using Infinispan as storage.

In my case i have comet users connected to specific Jboss nodes in cluster, have a separate queue for each comet client and some server-side API to send events to particular client. The problem is that Infinispan obviously doesn't have any iterator-style API and i've read some other discussions about the cause, so i want to try other approach taking assumptions from my case:

1. Create a separate cache for each connected client

2. Set separate consistent hash function for each cache so that all messages will be hashed to single node (to which comet client is connected)

3. Use distributed cache to achieve redundancy

4. On each node start poller thread which will take all comet clients connected to this node, iterate other them, for each client take corresponding cache, take keyset from it and iterate other it sending all messages to comet channel.

So the main idea is to store all messages of particular cache on a single node so that they will be accesible using keyset().

And here are my questions:

1. Does maintaining many (about 100 000) caches cause severe overhead? Will creating such "single-node hashed" cache be propagated to all nodes or it will only affect nodes which will contain values (is cache creation propagated to whole cluster or it is done lazily when first key is stored/requested on particular node)?

2. Is keyset() stable for described usage (what about transactions, XA, locks - any problems may arise?)? Docs say that it should not be used in production environment but forum discussions show that it is mainly because nobody though of appropriate usage of such thing.

3. Any other problems with described scheme?