-
1. Re: Data Distribution and Locality Issues
galder.zamarreno Jun 24, 2009 6:37 AM (in response to clajoie)Hi Clajoie,
We have a JIRA open to address these scenarios (https://jira.jboss.org/jira/browse/ISPN-78) but it needs further thought, design...etc.
If you have ideas on top of the ones mentioned in the JIRA, we'd be happy to hear them. And if you're interested in helping develop such feature, we'll be even more happier :) -
2. Re: Data Distribution and Locality Issues
clajoie Jun 24, 2009 8:21 AM (in response to clajoie)I think ISPN-78 is a bit different than what I was talking about. In my case the objects being stored in the cache may be relatively small but you might have a lot of them (i.e. an object is a few kilobytes but you have trillions of them).
As the owner of such a dataset you may want to treat it in a couple different ways. You may want to replicate it to a couple different geographically distinct sites (i.e. one copy in western US, one in eastern US, one in Europe, etc.). In this case you probably want all copies to appear as one logical copy so that if one site goes down the request for data is just routed to another site (though obviously data retrieval would end up being slower).
Alternatively you may want to "shard" the dataset so that records 1-1T go on cluster A, 1T+1 - 2T go on cluster B, etc. In this case you want the system to know that if you ask for item 1T+786 your request should go to cluster B and if you store entry 8758 it should be written to cluster A.
In either case you may want to be able to determine which site/cluster containing that data is "closest" (defined by some metric) so that you can then route some work to computing sources near that site/cluster instead of pulling millions - trillions of records back across the network.
Now, all of that said I'd very much like to help work on a problem like this. At the moment I do not know enough about the existing APIs and architecture but in 2-3 weeks I could probably provide you with a rough idea of how it might be possible to do such things. -
3. Re: Data Distribution and Locality Issues
galder.zamarreno Jun 26, 2009 3:29 AM (in response to clajoie)Hi, sorry I had misunderstood your original request.
Sharding of dataset is something we starting providing with Infinispan with the DIST or distribution mode and consistent hashing algorithm. You can already test this with the latest ALPHA5 release and remember that it includes a GUI demo that shows the distribution capabilities. More detailed information can be found in: http://www.jboss.org/community/docs/DOC-10278
What it's not possible at the moment is to force the replication group to have nodes that are in different geographic sites. For the moment, it will simply use the hash algorithm to find the nodes where to store data but it has no notion of location of these nodes or preferences. This is something that we might consider.
You could in theory however provide an implementation for org.infinispan.distribution.ConsistentHash in such way that keys 1-1T are directed to one node, 1T+1 - 2T to another...etc where these nodes are located in different geographic sites. However, currently this would require all sites to be part of the same cluster.
With regards to determining which node is "closest", in the client/server module (https://jira.jboss.org/jira/browse/ISPN-29) we're gonna be providing soon, the client will be able to apply the same algorithm used in the server and directly go to one of the nodes that contains that data. Again, for the moment this will not contain geographic information but we might accomodate it at some point. -
4. Re: Data Distribution and Locality Issues
manik Jul 6, 2009 1:34 PM (in response to clajoie)clajoie - this is definitely in the list of problems we want to solve. :) As Galder mentioned, DIST mode is what takes care of this (although I know the impl in ALPHA5 is not perfect and is pending some pretty severe improvements).
It is possible to "group" data items to be colocated on the same node. This can be done by providing your own hash algorithm (implementation of the ConsistentHash interface) with more specific knowledge of the objects being cached, and where they should be stored in relation to each other.
You should join infinispan-dev and join in the discussions!