Infinispan - DHCP renew IP address of node during run of cluster

[tomas11]

Hi all

I have following scenario. Cluster is build and setup. Than I force with DHCP one of the node to change its IP address. From that point it seems that cluster is split and I have following lines in logs:

ERROR Alarm-Type:0 [Timer-3,cluster,ac636131-a910-4c8e-9677-3645ab28969d-5264] (TP.java:1530) - JGRP000029: ac636131-a910-4c8e-9677-3645ab28969d-5264: failed sending message to 89379fdc-03da-4328-8eac-a3090ae1045d-26015 (170 bytes): java.io.IOException: Invalid argument, headers: RequestCorrelator: id=200, type=REQ, id=39, rsp_expected=true, UNICAST3: DATA, seqno=41, UDP: [cluster_name=cluster]

Is there a way how make cluster / nodes more robust so the change of IP address does not affect it and cluster continue works after the change of IP address normally?

Thanks

Tomas

[belaban]

Every cluster member is identified by a UUID that is associated with an address:port, at startup of the JGroups channel. If you change the address, that won't work anymore.

I suggest stop your Infinispan instance, change the address, then start it again.

[nadirx]

Hi Tomas,

I'm trying to figure out in which scenario changing the IP address makes sense for a server, but I can't think of one. Why are you doing this ?

[tomas11]

Hi Bela, Tristan

Thanks for feedback.

We have embedded Infinispan on every node. Situation can happen that during run of cluster IP address on one of nodes is changed - e.g. thanks to DHCP.

From what I read from Bela I'll guess that there is no option that cluster can handle such a change of IP address of one of cluster members?

There is need to restart Infinispan instance because of that?

[belaban]

Can't you setup DHCP such that a node keeps its address as long as there is activity? A change of address won't happen unless you have more nodes than available addresses competing for addresses between each other. In other words, you can prevent addresses being taken away from a node.

That said, you could inject the new address into JGroups programmatically, but you'd have to know that there was an address change in the first place.

[tomas11]

Hi Bela

How we can do the injection of address to JGroups programmatically? Does this action needs to be done on all nodes?

We did following:

1. Two nodes are running cluster

2. On node1 ip address is changed

3. node1 is disconnected from cluster (after IP address was already changed)

4. node1 is connected back to cluster (now with different IP address)

Result which we observed:

- from node1 we can see also node2

- but from node2 which stayed and held cluster - it can not see new node1

Is also some action needed to be performed on node2 (as a member which stayed in cloud)?

What are we missing to get to correct state? So that both node1 and node2 can see each other.

Thanks for help

[belaban]

The mapping from UUID to IP address is stored in the logical address cache in the transport (e.g. UDP or TCP). You'd have to update this table on every node, but this is a hack.

If you call JChannel.close() on node1 and then create a new channel and connect to the cluster again, this is a _new_ member so this must work (withough logical table change)!

I suspect you only did a JChannel.disconnect()/connect()?

[belaban]

The code below should work. Replace the array of addresses with your own addresses (and the config file) and start a few instances. When you press enter, a member will change its IP address (bind_addr).

I tried this with UDP, don't know if it would work with TCP as well.

public class bla extends ReceiverAdapter {

   protected JChannel ch;

   protected int current_index;

   protected final String[] bind_addrs={"127.0.0.1", "192.168.1.3"};

   protected void start(String name) throws Exception {

   ch=new JChannel("/home/bela/fast.xml").name(name).receiver(this);

  injectBindAddress();

   ch.connect("demo");

   for(;;) {

  Util.keyPress("enter to change address");

  System.out.println("disconnecting channel");

   ch.disconnect();

  injectBindAddress();

  System.out.printf("new bind_addr: %s\n", ch.getProtocolStack().getTransport().getBindAddress());

  System.out.println("connecting");

   ch.connect("demo");

  }

  }

   public void viewAccepted(View view) {

  System.out.printf("-- view: %s\n", view);

  }

   protected void injectBindAddress() throws Exception {

  String bind_addr=bind_addrs[current_index++ % bind_addrs.length];

   ch.getProtocolStack().getTransport().setBindAddress(InetAddress.getByName(bind_addr));

  }

   public static void main(String[] args) throws Exception {

   new bla().start(args[0]);

  }

}