Distributed cache xml example

[idonaldson]

Is there a complete example of a distributed cache being setup with strictly xml?

I have a need to have a high availability distributed cache setup across 6 VMs. I have three VMs to test my configurations with until I move to our production environment, so I've configured two servers for each running instance using the host-slave.xml file on VM2 and VM3 and the host.xml file on the domain controller VM.

VM 1:

Running as Domain Controller with domain.sh

Running server-one, server-two as hosts

VM 2:

Running server-three, server-four

VM 3:

Running server-five, server-six

Once I start the servers, using domain.sh for all three, passing host-slave.xml to VM2 and VM3, I can see them all join together and the configuration from the domain controller domain.xml is used on the other nodes, so the communication layer is working.

Questions:

1. When using the hotrod client, the data being put in the cache is not moved across the VMs, it only stays on the nodes that are on the same VM, what am I missing?

2. If I take the domain controller down and then bring it back up in a minute or two, server-three through server-six never rejoin the cluster from what I can see in the management console. I've read about nodes trying 10 times to connect, but what is supposed to happen after those 10 attempts? Does this require manual intervention to reconnect all the nodes?

3. Is there a Java hotrod client example for using a distributed cache? For instance, when setting up the RemoteCacheConfig I'm passing all three public IP addresses to the RemoteCacheManager, is this correct?

System:

Red Hat

Java 8

Hotrod 6.0.2

Infinispan 8.2.4.Final

[nadirx]

When you say "they join together", how are you determining that ? The thing that matters is the jgroups view. You should be seeing logs like the following:

[Server:server-two] 20:39:33,664 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (jgroups-8,master:server-two) ISPN000094: Received new cluster view for channel clustered: [master:server-two|1] (2) [master:server-two, master:server-one]

[Server:server-two] 20:39:33,667 INFO  [org.infinispan.CLUSTER] (jgroups-8,master:server-two) ISPN100000: Node master:server-one joined the cluster

in the end the final view should be something like [vm1:server-one, vm1:server-two, vm2:server-three, vm2:server-four, vm3:server-five, vm3:server-six]

What is the configuration of the cache ?

As for the Hot Rod client, if things are working correctly, it is sufficient to supply a handful of initial hosts and the client will retrieve all available servers on initial connection and receive topology updates if the cluster changes.

Tristan

[idonaldson]

Tristan,

When I say they join together, I'm speaking solely from looking in the web management console and I can see all the nodes listed under the cache.

Looking at the log snippets that you provided, when I restart a node it looks like the following:

[Server:server-four] 19:52:39,046 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000094: Received new cluster view for channel clustered: [host1:server-four|0] (1) [host1:server-four]

[Server:server-four] 19:52:39,059 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Channel clustered local address is host1:server-four, physical addresses are [172.XXX.XXX.XXX:55350]

Which to me looks like this server-four isn't getting connected to vm1, the domain controller.

The cache configuration is just a simple distributed cache that I added to the domain.xml.

I've attached the domain.xml, host.xml from the domain controller VM, and a host-slave.xml from one of the slave nodes. As well as a log capture from starting up the slave node.

Looking at the documentation again, how do I reference the default jgroups.xml as seen here: Infinispan 8.2 User Guide ?

Thank you!

Ian

[idonaldson]

Update time.

Turned off iptables on all VMs and VM2 and VM3 are forming a cluster with their nodes now.

[Server:server-five] 19:51:06,211 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000078: Starting JGroups channel clustered

[Server:server-five] 19:51:06,247 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000094: Received new cluster view for channel clustered: [host2:server-three|2] (3) [host2:server-three, host2:server-four, host3:server-five]

...

[Server:server-six] 19:51:10,057 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000078: Starting JGroups channel clustered

[Server:server-six] 19:51:10,095 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000094: Received new cluster view for channel clustered: [host2:server-three|3] (4) [host2:server-three, host2:server-four, host3:server-five, host3:server-six]

However I can't get the nodes that are on the same VM as the domain controller to join the rest of the group. I've ran the JGroups McastSendTest and McastReceiveTest in all different configurations and it's all working at that level.

Not sure where to go from here?

[nadirx]

Hi Ian,

I guess we'd need some more logs from the startup. Ideally the log from each node, but I guess one node per host could suffice.

[idonaldson]

Tristan,

I cleared the logs and restarted all the services. The attached is the server.log from each machine as well as all the configurations from each.

Thanks again for your help with this.