Infinispan - cross site deployment with 400 nodes

[horneastefan]

Hi Guys,

We are currently evaluating the possibility of using Infinispan as storage for a service registry.

Our required deployment will be like this:

- running 20 Infinispan instances that will communicate amoung each other on WAN. Let's call these type A nodes

- for each type A node, there will 20 type B infinispan nodes. (So, 400 B nodes). The 20 B nodes that are children of the same A node will speak amongst each other on a WAN.

- every B node will contain a map of (serviceId, serviceStatus) pairs. The service status value will be updated at a frequency of less than 30 seconds.

- every B node should have locally the service data (serviceId - serviceStatus pair) for all other 400 B nodes, not just for their siblings. The data should be persisted locally(maybe by using the Single File Store, or a DB Store).

- We were thinking that all B nodes that are children of the same A node to communicate between each other to exchange serviceID - serviceStatus info and then to pass it to node A, that will replicate that data to all other 19 nodes A.

- Node A will send  service data for node B's that are not his own children to its children.

- Each Node B will manage 10 services, so the total number of keys in the map is 4000.

I have depicted below the proposed deployment, reducing the number of nodes and communication paths (all nodes A communicate between each other, and all nodes B communicate with each  other sibling and with the parent node A).

Is replicated mode combined with cross-site replication a solution to these requirements? I noticed that the docs are saying that replicated mode works on max 10 nodes, but I was hoping it will work in our scenario, as we have little data to store in Infinispan. Also, there is no number of supported sites in the cross site replication docs.

I am also following this somehow related thread: Weighing Replicated Mode Against Distributed Mode

Cheers,

Stefan

[pruivo]

Hi Stefan,

why is every node communicating via WAN? is every node in its own data center?

Replicated mode works with any number of nodes but it is not cheap. The higher the number of nodes involved in a replicated cache, the high the response time in each update and that is why we don't recommend replicated mode for large clusters.

The cross site replication also work fine with any number of nodes but, in the same way as in replicated caches, the communication cost increases when multiple sites are involved.

There is a small issue: the backup site won't backup remote data to another site. For example, if you have Site1 backups to Site2; and Site2 backups to Site3; if you update data in Site1 it is going to be replicated to Site2. Site2 won't replicate to Site3!

Why the nodes B needs to have the data locally?

I think a client-server approach would fit better in your scenario. You could create a cluster of A nodes with Infinispan Server and the B nodes would be the clients.

The clients may store some data locally with near cache - check Infinispan 9.0 User Guide.

Cheers,

Pedro

[horneastefan]

Hi Pedro,

Thanks for the quick reply.

Regarding WAN Communication, as a general rule, all nodes are in their own data center, and only in some isolated cases a data center will host more than 1 node.

I will think if we really need to have all the data on each B node. Maybe between B nodes a partitioned cache could work, and we should only store the whole data locally on node A. On node A the data should be stored as service data will contain personal contact information for all admins of nodes A and B, so in case of a partition where node A cannot comunicate with any other node, the admin of that node A can at least obtain the contact information of other admins from it's own locally stored data. (Of course it would be ideal that a node B admin should have access to that info, we will figure out some way for node B admins to access the whole data found on the parent node A).

I will study the user guide for client server deployment and will get back to you.

Cheers,

Stefan