5.1.0.0 Beta 4: Problem Synchronizing HotRod Distributed Cache Nodes

[fealves78]

I am trying to load about 1000000+ records from a database into a distributed HotRod cluster using Infinispan 5.1.0.0 Beta 4. All the nodes seem to be correctly configured. I am able to start up all nodes and connect to the cluster using a very simple remote client application. However, a few minutes after the loading process was initiated, I start to get error messages in the individual console for each HotRod node.

Also, before I start getting these errors, if I turn off one of the nodes while the data is being loaded into the cluster, the whole cluster gets really busy. Lots of error messages start to show up and in some point I received the message "disk full". The problem gets worst if I am using Async Calls with this mode.

To me it seems that the cluster is having problems in rehashing keys and keeping the nodes synchronized while dumping a lot of data into the disk.

I ran the same tests in replication mode and I got no problems at all. I was able to load all the data into the cluster while switching some of the nodes on and off, and the cache responded very well throughout the whole process. Unhappily, the same did not happen in distribution mode.

I am attaching my configuration files to this post. Please let me know if there is anything I can do to improve the configuration settings for the distribution mode.

I thank you in forward for your help.

[fealves78]

I also would like to add that the same tests were performed with Infinispan 5.1.0 Beta 5 and the same faulty results were obtained.

[galder.zamarreno]

What errors are you getting?

Please attach the client code used to load the data.

There's a few things that you can maybe try:

1. Disable L1 to avoid overloading a node with data that it doesn't own

2. Enable virtual nodes to make distribution more equal. It could be that one node is taking a lot more data than others which causes issues to the rest of nodes. Configure for example, 1000 virtual nodes (numVirtualNodes attribute in hash - by default this is 1)

[fealves78]

Could you please let me know how can I enable virtual nodes and in which file I should add this configuration to?

I will redo the tests and produce a new error log to send to you (it will be a huge file though due to my current use cases).

Thank you for your help.

[galder.zamarreno]

Galder Zamarreño wrote:

... Configure for example, 1000 virtual nodes (numVirtualNodes attribute in hash - by default this is 1)....

Your infinispan config files:

<hash numOwners="2" numVirtualNodes="10000"/>