Data grid platform use cases / programming idioms?

[triforkpol]

Hi,

I am actively seeking out which programming/design idioms or use cases that a data grid like Infinispan can be used to solve. As such, it is not so much Infinispan specificly, but the data grid platform type of products out there as a whole. I have a hard time finding any information on this. The documentation of Infinispan itself says much about what Infinispan can do technically - distributions, txs and stuff like that - but not much (anything?) about in which situations a data grid is a viable and good alternative to say, the typical RDBMS solution.

For instance. Given I have a huge table - which I do have :-) - which is to be queried very frequently by many clients. So much, that my RDBMS will suffer (suffers) from it. Would it be a viable alternative to simply cache this database table into Infinispan, all of it, and then scan/search using the query api instead?

When simply looking at the data grid apis out there, they look like a Map. And a Map performs well, when I know the key. If I go about scanning entries in the grid, I would think to suffer great in performance of such queries.

Am I going to rethink how I key stuff in a grid, so that I for instance build a tree (or many trees), so that I can go to the same actual data in the grid through different keys? To me, that seems like starting to implement in my own code, what a RDBMS have done for me (trees, indices, etc.). I wouldn't want that.

Do you guys have any input on this? Either specificly on my example above, or maybe pointers to articles or books about the subject in general. With the cloud coming (already here?), I would say this is an area of systems design, that we need to understand. I do not understand the use cases for it myself (except from the simple one: A very large, fast and scalable cache).

?

/Per

[mircea.markus]

This is a very interesting question.

One of the problems that a data grid solves is (virtually) linear scalability: adding N% more (commodity) servers will allow the systemt to support N% more load with the same read/write response time. Ths is achieved through data distribution/partitioning, and the only limiting factor is the network which is shared by all nodes. Network can be quite 'scallable' though, e.g. gigabit ethernet. I would say this by itself is a big gain, as RDBMS scalability problems are not something trivial to solve.

Regarding the querying, besides the map key querying, we have in plan an JPA like API that would allow one to perform queries in an JPA-like fashion: https://jira.jboss.org/jira/browse/ISPN-24.

Another use case I can think of is for highly transactional applications, as handling the data in memory is much faster.

[manik]

Infinispan also does offer querying. See http://community.jboss.org/wiki/queryinginfinispan - this is just a "tech preview" at the moment and the API isn't finalised, but this is something you can try out, so you don't have to scan the data grid for entries.

[triforkpol]

Hi Manik,

I actually have tried it out. I was looking for a more broad view / information on programming models with data grids, as such. Not so much the technical details of a specific grid.

Anyways, I guess this sort of thing is something that will emerge, when (if) the masses actually start using the cloud and data grid solutions in it. Stuff like design patterns for using data grids, does and dont's etc. Stuff that we have literally TONS of literature on for something like RDBMS's.

/Per

[manik]

Anyways, I guess this sort of thing is something that will emerge, when (if) the masses actually start using the cloud and data grid solutions in it. Stuff like design patterns for using data grids, does and dont's etc. Stuff that we have literally TONS of literature on for something like RDBMS's.

Yeah this sort of thing will come with maturity of this space.  NoSQL, data grids, key-value and column oriented stores are all pretty nascent still.  Great potential to help define those standards/patterns/idioms though!