4 Replies Latest reply on Jul 6, 2009 1:34 PM by manik

    Data Distribution and Locality Issues

    clajoie

      In some of the projects I work with we deal with pretty massive data sets (usually measured in petabytes). In cases like this the data is often distributed in certain locations. Sometimes this distribution is via replication and at other times it is done more similarly to database sharding. In either case a lot of effort goes towards ensuring that the data appears to be in just one place (regardless of outages and the like).

      I was wondering if you were considering these types of uses cases within the Infinipsan architecture. It seem like, from a high-level, such support would require intelligent data management (command routing, distribution, etc). Additionally, exposing locality information (i.e. answering the question "where are data X, Y, and Z?") would allow an application to route work to computational resources "near" the data.

      I know the question is pretty broad. I was just curious if there has been any thinking along these lines.

      Thanks!

        • 1. Re: Data Distribution and Locality Issues
          galder.zamarreno

          Hi Clajoie,

          We have a JIRA open to address these scenarios (https://jira.jboss.org/jira/browse/ISPN-78) but it needs further thought, design...etc.

          If you have ideas on top of the ones mentioned in the JIRA, we'd be happy to hear them. And if you're interested in helping develop such feature, we'll be even more happier :)

          • 2. Re: Data Distribution and Locality Issues
            clajoie

            I think ISPN-78 is a bit different than what I was talking about. In my case the objects being stored in the cache may be relatively small but you might have a lot of them (i.e. an object is a few kilobytes but you have trillions of them).

            As the owner of such a dataset you may want to treat it in a couple different ways. You may want to replicate it to a couple different geographically distinct sites (i.e. one copy in western US, one in eastern US, one in Europe, etc.). In this case you probably want all copies to appear as one logical copy so that if one site goes down the request for data is just routed to another site (though obviously data retrieval would end up being slower).

            Alternatively you may want to "shard" the dataset so that records 1-1T go on cluster A, 1T+1 - 2T go on cluster B, etc. In this case you want the system to know that if you ask for item 1T+786 your request should go to cluster B and if you store entry 8758 it should be written to cluster A.

            In either case you may want to be able to determine which site/cluster containing that data is "closest" (defined by some metric) so that you can then route some work to computing sources near that site/cluster instead of pulling millions - trillions of records back across the network.

            Now, all of that said I'd very much like to help work on a problem like this. At the moment I do not know enough about the existing APIs and architecture but in 2-3 weeks I could probably provide you with a rough idea of how it might be possible to do such things.

            • 3. Re: Data Distribution and Locality Issues
              galder.zamarreno

              Hi, sorry I had misunderstood your original request.

              Sharding of dataset is something we starting providing with Infinispan with the DIST or distribution mode and consistent hashing algorithm. You can already test this with the latest ALPHA5 release and remember that it includes a GUI demo that shows the distribution capabilities. More detailed information can be found in: http://www.jboss.org/community/docs/DOC-10278

              What it's not possible at the moment is to force the replication group to have nodes that are in different geographic sites. For the moment, it will simply use the hash algorithm to find the nodes where to store data but it has no notion of location of these nodes or preferences. This is something that we might consider.

              You could in theory however provide an implementation for org.infinispan.distribution.ConsistentHash in such way that keys 1-1T are directed to one node, 1T+1 - 2T to another...etc where these nodes are located in different geographic sites. However, currently this would require all sites to be part of the same cluster.

              With regards to determining which node is "closest", in the client/server module (https://jira.jboss.org/jira/browse/ISPN-29) we're gonna be providing soon, the client will be able to apply the same algorithm used in the server and directly go to one of the nodes that contains that data. Again, for the moment this will not contain geographic information but we might accomodate it at some point.

              • 4. Re: Data Distribution and Locality Issues
                manik

                clajoie - this is definitely in the list of problems we want to solve. :) As Galder mentioned, DIST mode is what takes care of this (although I know the impl in ALPHA5 is not perfect and is pending some pretty severe improvements).

                It is possible to "group" data items to be colocated on the same node. This can be done by providing your own hash algorithm (implementation of the ConsistentHash interface) with more specific knowledge of the objects being cached, and where they should be stored in relation to each other.

                You should join infinispan-dev and join in the discussions!