Intra-cluster communication protocol

[zam0th]

Just a quick question to confirm: what is the protocol for communication between cluster nodes? I understand that JGroups is used for cluster discovery, HotRod (and variety of others) is used for client/server communication, but i couldn't find anything in the docs about how cluster nodes communicate between each other for e.g. cache replication? What's the protocol difference between "sync" and "async" replication modes: is that TCP in the former and UDP in the latter, or something else? How can i fine-tune this protocol or even do basic configuration for it?

[nadirx]

JGroups is used for all intra-cluster and cross-site communication.

JGroups handles all TCP/UDP communication, discovery, flow control, message fragmentation, cluster security and failure detection.

Infinispan uses JGroups messages to deliver its commands to the cluster nodes via a component we call Transport.

Sync and async replication are higher-level constructs that have little to do with the underlying transport.

So, when you mention fine-tuning, it depends whether you are concerned about the networking aspect or the cache replication aspect, but you can do both.

For example you'd tune JGroups to change the discovery protocol and timeouts, the failure detection algorithms and timeout, the flow control credits, the TCP/UDP buffer sizes. See JGroups 4.x's protocol list.

To tune Infinispan's clustering configuration you can do it per-cache: ClusteringConfigurationBuilder (Infinispan JavaDoc All 9.4.10.Final API) . Here you can set replication timeouts, state transfer details, etc.

If you have a specific need, ask here

Tristan

[zam0th]

Yes, i understand all that, but what is the protocol to transmit data itself, not control messages? E.g. when you modify contents of a cache, you need to replicate the data, how is it done? You can't do that with multicast, can you?

[nadirx]

It's all represented as Infinispan commands wrapped into a JGroups message.

For example, the command that corresponds to cache.put(k, v) is implemented by the following:

https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/commands/write/PutKeyValueCommand.java#L79

Commands are then sent to other nodes by org.infinispan.remoting.rpc.RpcManager. If a command is sent to all nodes, JGroups will use multicast if you are using UDP to send it, otherwise it will use unicast.

[zam0th]

nadirx  wrote:

JGroups will use multicast if you are using UDP to send it, otherwise it will use unicast.

Oh, i see now, thank you! But how is that determined, i mean the "knowledge", the default configuration stating that syncrounous replication must use unicast and not anything else? Or is there anything else ?

[nadirx]

If you use a TCP transport, it will all be unicasts.