How to ensure failover of Domain Controller?

[udit-mishra-5113a21a]

When Host Controller process gets started, it knows which DC to connect through the <remote> tag under the <domain-controller> tag in hosts.xml file. Suppose if the DC is down, HC tries for some 10 times, and then I see Press any key to continue..

Keeping these points in mind, I have some questions -

1. Can I have multiple DC?
2. Is there any way I can ensure that if one DC is down it should try to connect to other DC? I am taking about HC to DC communication, not client to cache server
3. Can I use Apache for such redirection?

[nadirx]

First of all, remember: a domain controller is not essential for a cluster to serve requests. It is only required if you want to perform management operations.

My suggestion is to place the DC on a virtualized host that can be recreated anywhere.

Additionally the domain.sh / domain.bat script has two switches that can help you out:

--cached-dc                         If this host is not the Domain

                                        Controller and cannot contact the

                                        Domain Controller at boot, boot using a

                                        locally cached copy of the domain

                                        configuration (see --backup)

--backup                            Keep a copy of the persistent domain

                                        configuration even if this host is not

                                        the Domain Controller

[udit-mishra-5113a21a]

Okay, its not essential for serving request. But I think it is also required when initially the cluster is initializing; otherwise how would HC reads the configuration from DC, that is why initially after some try HC stops. Although if once initialized properly, it continues to run, yet it will not be able to perform management operations, as you said.

Regarding the second point, I tried that, by entering to the bin directory, opening command prompt and by typing -> domain.bat --cached-dc --backup (enter), but it threw these errors -

Boot Thread) WFLYHC0149: Option --cached-dc was set; obtaining domain-wide configuration from domain.cached-remote.xml

[Host Controller] 11:01:21,179 WARN  [org.jboss.as.host.controller] (ControllerBoot Thread) WFLYHC0031: Cannot load the domain model using using --backup

[Host Controller] 11:01:21,245 ERROR [org.jboss.as.host.controller] (ControllerBoot Thread) WFLYHC0008: Failed to start server (ServerTwo): java.lang.IllegalArgumentException

So, I am certainly missing something. Where this domain.cached-remote.xml will be located? Do I have to copy the domain.xml from DC and paste it into HC and rename it to this file?

Apart from this, one more thing I found was that I can write <discovery-options> tag inside <remote> tag in hosts.xml file. I tried that, It worked also. I had two DC and one HC, all three had server node(so total 3 nodes). I started both DC1 and DC2 and then HC1, it got connected to DC1. Then I turned down DC1, HC retried and got connected to DC2 then. What If I make configuration changes in DC1 through Web Administration Console, like Adding a new cache. It functions fine, but how will it get replicated to DC2 ? Or it won't?

[nadirx]

The --cached-dc and --backup options are mutually exclusive. First of all use --backup to create a local copy. Then use --cached-dc to tell a HC to use the backup if it cannot contact the DC.

As for the second part, it would be up to you to ensure that all DCs get a copy of the configuration.

[udit-mishra-5113a21a]

I followed what you said exactly.

I first ran it with --backup option and it copied the domain.xml file into new domain.cached-remote.xml file

Now I stopped both DC and HC and then I started HC alone, it retried to connect to DC for some 10 times and then it could not start, exactly same behavior which was earlier. So, what difference did the backup made?

[udit-mishra-5113a21a]

Hey nadirx , please help me out. I really need to know, why so?

[udit-mishra-5113a21a]

It was my mistake again,configuration issues were there. It worked fine.

Thank you nadirx for helping us out

[stejas7]

Hello nadirx,

Currently i am using infinispan 8.2.3. when i am start dc in cmd with domain.bat --backup tag. its doesn't create a domain.cached-remote.xml file .

[poojakatkade]

Hi Udit,

I am trying same thing on wildfly16.0.0 . can you plz elaborate what changes you have done in primary domain controller and in backup domain controller?

[jafarre.viewnext]

I have to DC's running (actually, one DC and one slave ready to become the DC if it's necessary).

The setup is as follows:

- One main (master/active) DC

- A second (slave/inactive) host controller started with --backup option. It keeps a copy of the full configuration and acts as a second inactive DC which can be manually turned into the active DC.

- There must never be two active DC's at the same time

- Each host controller connecting to the DC is run with the --cached-dc option. This keeps a copy of the part of the configuration which is relevant to that host controller, and allows it to run even if the DC is down.

- Each host controller connecting to the DC specifies the two (possible) DC's in static discovery.

- In case of failure of the DC, first make sure the main DC is down or stop it. Then turn the backup DC into the main DC (see the link below).

For more information:

Chapter 8. Domain Management Red Hat JBoss Enterprise Application Platform 7.2 | Red Hat Customer Portal

It is possible to setup Apache httpd load balancer to use only the main DC and keep the other one as failover.

Please, see the details here:

https://httpd.apache.org/docs/2.4/howto/reverse_proxy.html

Kind regards.

[galder.zamarreno]

You should ask this in the EAP or WildFly forums really.

If using Infinispan domain mode, this is going away btw. In Infinispan 10 we're no longer based on WildFly/EAP.