Message bundling and cluster RPCs

There is a problem if messge bundling is enabled and we invoke a lot of clustered RPCs (e.g. through JGroups' RpcDispatcher or JBoss' ClusterPartition, or JBossCache): since we wait for N milliseconds or M bytes to accumulate, when we have many small messages that are created as a result of invoking a cluster RPC, then we incur the wait time of N.

The result of this is that one simple cluster RPC has to wait for N milliseconds (e.g. 30ms in the default), before the call is sent to the cluster. For each response the receiver also waits for 30ms max, so the total wait time for one cluster RPC could be as much as 30ms for the request and 30ms for the response(s): 60ms ! This would allow us to invoke ca 16 cluster RPCs per second  (This doesn't even take method call handling at the receiver and request / response correlation at the sender into account).

The impact of bundling is currently under investigation here .

TCP and Nagling

TCP by default nagles messages, that is, conceptually, smaller messages are bundled into larger ones. If we want to invoke synchronous cluster method calls, then we need to disable nagling in addition to disabling message bundling. This is done by setting TCP.tcp_nodelay="true".