How does JBoss Cache perform with high (15-20) number of jvm

[souravm]

Hi,

Has anyone so far used JBoss Cache with high number of jvm instances like 15-20 ?

We are planning to use jboss cache with replication (asynch or synch) for 15 - 20 jvm instances.

There is a scenario where new data would be created every day during first time login of the users at the start of the day. That point of time every jvm instance would get new data and would try to replicate it to others - that way giving rise to a situation like 15x15 replications. Just wondering how much CPU cycles would be consumed in marshaling/unmarshaling as well as the n/w bandwidth.

The data size won't be much high - around 1KB per data getting replicated in other instances. However, in a second 20-25 such data would get created.

Any insight/input in this regard is appreciated.

Regards,
Sourav

[belaban]

So I'm assuming we have a 20 node cluster, and every node sends 25KBytes/sec (25 KBytes == 200KBits).

When using UDP, every node sends 1 multicast packet, to be received by every other node. So you scenario above (15x15) doesn't apply.

When using TCP, every node sends a message N-1 (19) times, so your scenario *does* apply in this case.

With respect to network bandwidth, sending is not the issue because every node has a full duplex line to the switch.

However, when receiving, a line from the switch to a node has to be shared between 19 (N-1) instances. In other words, every node can receive concurrent traffic from the 19 other nodes.

So if you have a 1GB switch, then the effective rate / node is 1000MBits/19 = 52MBits = 52'000KBits = 6'500KBytes /sec. That 260 times more that you need !

So this is peanuts traffic-wise. The bottleneck might lie somewhere else, namely in the data: when everyone replicates its date to everybody else, every node has to store DATA * 20 on average. So if every node has data of 1MB, then the avg data size on a node is 20MB. This is fine, but of course not scalable if (1) your avg data size increases or (2) your cluster size increases.

[souravm]

Hi Bela,

Thanks for your inputs.

Yes I agree with you, n/w bandwidth wise it may not be an issue.

However, I've not got your point on how data store can be issue w.r.t scalability. Let me explain the data model a bit more here. All together I've around 250x1000 such entities, which would be replicated across 20 nodes (in about 2-3 mins timeframe). Now each entity being a size of 1 KB, we have provisioned for 250MB memory at each instance level (that means every node has all the state of the system). Given that I think with increase in number of instances in the cluster, the memory need would not increase. Please let me know if you think otherwise. However, I agree increase in average data size would surely be a scalability issue.

My concern is more with the CPU cycles which will go in marshaling/unmarshaling data. Given that every instance would receive data from all other (N-1) instances, the CPU cycles in marshaling the data would grow in all nodes in addition of more nodes in the cluster.

Also I'm not sure how it would impact the locking. Assuming that use of MVCC locking strategy would take care of this.

By the way, I'm interested to know when is the target date of release of consistent hash algorithm (CHA) based (select where keys reside) distributed caching mode (ISPN-30). That would surely solve the scalability problem what I see in this case.

Regards,
Sourav

[belaban]

"souravm" wrote:
All together I've around 250x1000 such entities, which would be replicated across 20 nodes (in about 2-3 mins timeframe).

Do you mean that every node has to store 250MB of data ? By 'replicated' I infer total replication: everyone replicates to everybody else...

Now each entity being a size of 1 KB, we have provisioned for 250MB memory at each instance level (that means every node has all the state of the system). Given that I think with increase in number of instances in the cluster, the memory need would not increase.

OK, but then you're not using total replication, I assume 250MB is the total data distributed across the cluster ?

My concern is more with the CPU cycles which will go in marshaling/unmarshaling data. Given that every instance would receive data from all other (N-1) instances, the CPU cycles in marshaling the data would grow in all nodes in addition of more nodes in the cluster.

Yes. Note that ispan doesn't use java.io.serialization though, so this is pretty efficient. But, yes, it might still be a perf issue.