Wildfly10 instance configured as replication-slave could not deploy queues

[wwang2016]

Hi

I configured two wildfly 10 instances with activemq configuration setting one being replication-master and the other being replication-slave.

The following were what happened:

(1) start up wildfly10 instance configured as replication-master first, then start up wildfly10 instance configured as replication-slave. There is no issue

(2) start up wildfly10 instance configured as replication-slave first, the instance does deploy queues and it is not started up properly.

Is this expected behavior?

Thanks

[jbertram]

Yes, this is the expected behavior as noted in the documentation:

...if a [shared-store] backup starts and does not find a live server, the server will just activate and start to serve client requests. In the replication case, the backup just keeps waiting for a live server to pair with. Note that in replication the backup server does not know whether any data it might have is up to date, so it really cannot decide to activate automatically. To activate a replicating backup server using the data it has, the administrator must change its configuration to make it a live server by changing slave to master.

[wwang2016]

Hi Justin,

I am working on setting up the whole application as HA singleton provider and want to ensure HA for messaging.

Can I configure both instances as replication-master? Is that a proper approach?

I noticed that in shared-store approach, I actually configured both instance as shared-store-master. Either one can start up properly. I guess normally you would configure one being master, and the other being slave?

Thanks,

Wayne

[jbertram]

Can I configure both instances as replication-master? Is that a proper approach?

If you have two instances and they are both configured as replication-master then where will they replicate to?  There has to be a slave to receive the replicated data, and without the replicated data there is no HA.

I noticed that in shared-store approach, I actually configured both instance as shared-store-master. Either one can start up properly. I guess normally you would configure one being master, and the other being slave?

In the shared-store configuration the slave will activate fully even if it can't find a live server (as noted in the documentation I quoted above).  The recommended configuration is master-slave in either the shared-store or replication use-case.  Using master-master may technically work in some scenarios, but that's is not considered a valid configuration.

[wwang2016]

Hi Justin,

I checked my shared-store configuration. It actually configured two activemq servers with one being shared-store-master and the other being shared-store-slave (pointing to another wildfly instance). The following is the configuration:

wildfly 1 (master + slave)

wildfly 2 (master + slave)

The activemq directory of wildfly 1.master is referenced by wildfly2.slave and the activemq directory of wildfly 2.master is referenced by wildfly1.slave

I am not setting up the traditional cluster where both wildfly instances are up and running, instead,I am setting up HA singleton so that only one wildfly instance is actively serving request and the other wildfly instance is in standby mode. When I test HA singleton scenarios (starting w1, w2, shutting down w1, restart w1, shutting down w2, restarting w2 ). Everything looks fine. So this setup seems to work for HA singleton and HA for messaging.

The master-slave configuration with data-replication has issues with the additional configuration in my case: HA singleton for the application. For example, if the slave instance is started up first, then the slave is HA singleton provider, but the activemq server does not get activated, and it waits for the master instance. This will cause outage until the master is started up. However, the master will not get activated either since the slave was started up first and is the current HA singleton provider, but not working. So then none of them is working until I shut down the slave instance.

Basically, as long as the slave instance is started up without the master instance started up first, this configuration will not work with the additional HA singleton configuration.

I check the document and found the colocate option which may be set up for data replication approach. Is this potentially a solution?

Due to the additional requirement on HA singleton, both instances should be working like mirror image without one being dependent on another.

Thanks,

Wayne

[jbertram]

With colocated approach (where both nodes have a master + slave) then the broker will be active on both nodes.  I'm not sure if that's the approach you want since you're going for an HA singleton.  The difficulty you're facing is because you're mixing the application server's implementation of HA singleton with the messaging broker's implementation of HA and the messaging broker doesn't have a notion of HA singleton.  You might be better of splitting your messaging broker from the application server so each can be configured as necessary.  If that's not an option then you should probably stick with shared-store HA rather than replication since it has the problem where the slave can't be started without its master.

[wwang2016]

Hi Justin,

I understand the reason why the slave instance does not deploy queue when the master is not up.

Just curious about if there is a way to allow the slave instance to activate activemq when the master is not up and running? This seems to be a quick solution to my specific scenario

Thanks,

Wayne

[jbertram]

Just curious about if there is a way to allow the slave instance to activate activemq when the master is not up and running?

In short, no.  The slave instance cannot activate because it doesn't know if its data is up-to-date until it can connect to the master.  I noted this in a previous comment already.  To allow the replicated slave to start without first contacting the master would be to risk compromising data integrity which is exactly what the slave exists to prevent.

[wwang2016]

Got it

Thanks

Wayne