During first start set node identfier to generated UUID

[mnovak]

Hi,

when I started WF10/EAP 7 then I saw following warning in server.log:

11:39:12,686 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 70) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.

From user perspective it doesn't look good when there are warnings in log during start of clean server and it's annoying going to config and set it.

It would be great to avoid it. What about to set node identifier to generated UUID which would be stored in Arjunas journal when WF10/EAP 7 is started for the first time. All other starts would load node identifier from Arjunas journal. Of course if node identifier is set in configuration then UUID would not be generated. This would allow to get rid of this warning. This is basically how for example Apache Artemis is setting its broker id. wdyt about this idea?

Thanks,

Mirek

[zhfeng]

Hi Miroslav,

I think we had the similar issue before [WFLY-1119] Assign an unique NodeID automatically - JBoss Issue Tracker and we rejected it as the fixed length of node is 28 and we have no way to check if it is unique. Also the user should be forced to do this because it carries risk.

Thanks,

Amos

[tomjenkinson]

+1 - although it might be possible to utilize this if all node shared access to the same JDBC store? That might be a useful facility for OpenShift for example.

[mnovak]

Thanks Amos - looks like this idea is not new at all.

I'm not sure what the max node identifier size is  - is it 28 bytes? UUID has 16 bytes (128 bits). Still if it's too long then there is a possibility to generate shorter "UUID" for example by generating random byte array (or use some other method). It will not be so "strong" as UUID but still good enough.

I can see that currently node identifier is set to 1 in WF10/EAP 7:

"node-identifier" => "1"

To continue with discussion on WFLY-1119 - it looks like that users don't fully understand importance of node identifier and what are the consequences of having duplicated node identifiers. Generating it random would avoid 99,99...% of user issues with this. We can still keep the warning to set it unique but as precaution let's set it random.

Tom has a very good point - EAP7/WF10 must work on OpenShift 3. It's using docker containers with EAP 7 to scale up. Those docker containers are all the same with the same configuration. This would allow not to bother by setting unique node identifier.

wdyt? :-)

[zhfeng]

The node identifier defines as the following.

public static final SimpleAttributeDefinition NODE_IDENTIFIER = new SimpleAttributeDefinitionBuilder(CommonAttributes.NODE_IDENTIFIER, ModelType.STRING, true)
            .setDefaultValue(new ModelNode().set("1"))
            .setFlags(AttributeAccess.Flag.RESTART_ALL_SERVICES)
            .setAllowExpression(true)
            .setValidator(new StringBytesLengthValidator(0,23,true,true))
            .build();

So it is a string with the max length is 24.  We use it in the Xid (which imposes a 64 byte limit). The "UUID" convert to string looks like 32 bytes.

UUID.randomUUID().toString.replace("-", "");

It is possible to substring the first 24 bytes as the node name. It could be a useful feature for working in the openshift. So tomjenkinson, we might consider reopen the WFLY-1119

[tomjenkinson]

Sure - let's reopen but it shouldn't be the default which therefore won't help Mirek with his warning.

[mnovak]

Thanks guys. I think setting random generation is better than current default which is setting "node-identifier=1". It reduces risks that there were will be duplicated node identifiers to almost 0. We can still have warning like: "node-identifier was generated randomly to "some generated id" - make sure it's unique in your environment" or something like this.

[tomjenkinson]

So long as it shows a WARN that is OK. The issue with storing in the object store though is that someone can delete it accidentally whereas normally the node id is in the config file.

[mnovak]

If id was generated and some one deletes it from object store then new one would be generated. This is something very unfortunate. Normally the whole JBOSS_HOME/standalone/data directory with objectstore would be deleted to clean the server. I'm not sure if any guarantees can be made in this case.

Maybe WF11/EAP7 could generate id and store in configuration. Integration layer for transaction subsystem could actually generate this id and store in configuration. But I think this would be a double safety bonus feature as it's very unlikely to happen.

[ochaloup]

I think this could be a nice feature. It would be good to ease the life of the user. I think there is a good chance that node id will not be changed by administrator even in bigger environment, it could happen when WFLY/Swarm is used in microservice world where multiple containers are started, it happens for sure in case user does not understand the meaning of the warning (that's whoever comes the first time to use Narayana doesn't) etc.

Regarding of the uniqueness, having 28 characters taken from a random generator is a quite good chance not hitting the same one on two nodes. What do you think?

mnovak would you have a link to Artemis code where the id is generated? I think it could be a good to get inspiration.

zhfeng I'm sorry for my ignorance but why the maximal length of 24 utf characters could be problematic?

tomjenkinson I think so. It sounds beneficial. Do not Openshift set this already somehow? Using the same node id for Narayana could already bring troubles. It could be fine for recovery process too. If node id is saved in object store then if WFLY node crashes it is ok to start another node just pointing to the existing object store and recovery would grab the node id to finish in-doubt txns (I see there's the limitation of having the same resources configured).

[ochaloup]

I was searching what's the way how the Openshift configures and I can see that it set the node id to {{jboss.node.name}}

https://ce-gitlab.usersys.redhat.com/ce/jboss-dockerfiles/blob/develop/scripts/os-eap7-openshift/added/standalone-openshift.xml#L417

and that's set by sh script

https://ce-gitlab.usersys.redhat.com/ce/jboss-dockerfiles/blob/develop/scripts/os-eap7-launch/added/launch/ha.sh#L40

Still, I wonder if it's possible to reconnect to an existing object store when a node crashes and having the new node having set the node id of the old node.

[mnovak]

Artemis has utility class for generating UUID:

https://github.com/rh-messaging/jboss-activemq-artemis/blob/jboss-1.1.0-x/artemis-commons/src/main/java/org/apache/activemq/artemis/utils/UUIDGenerator.java

and is used during start of Artemis server here:

https://github.com/rh-messaging/jboss-activemq-artemis/blob/master/artemis-server/src/main/java/org/apache/activemq/artemis/core/server/NodeManager.java#L208

First it tries to load node id from a file ".../data/activemq/journal/server.lock". If node id is not there then it's generated ans stored to this file. If node id is there then it just set nodeId of starting Artemis broker during start.

[tomjenkinson]

If the server has no configuration to distinguish it from the other servers and it connected to a shared JDBC store it would read the same node identifier as other servers which would not be good.

What you are describing could be done (entirely in WildFly code as it is responsible to set the node identifier: wildfly/CoreEnvironmentService.java at 6b61a6003f704221f66dcd9f418bcb7af88fb9ab · wildfly/wildfly · GitHub ) but I would not recommend it. Node identifiers are better set by an administrator to be certain that they are configured in a known manner. It is entirely possible that the administrator follows a process as you defined but putting that into the library seems dangerous as Narayana does not know its operating environment (e.g. does it share an object store).

[mnovak]

If one object store can be shared by more TMs as it's in shared JDBC store then it does not make sense to store node id in object store. It seems that only improvement can be done on transaction integration layer in WF/EAP that it will generate its node identifier and store in configuration of WF/EAP. Other starts would read it from configuration and not generate it. But I'm not sure if it's feasible.

[tomjenkinson]

Yeah, that would be good but I would still think it might fail the "openshift use case" - potentially those servers are provisioned using docker (?) which might reset to the SHA just after WFLY is unzipped and so lose the step where a node id is allocated?

What might work (based on your suggestions) is:

1. WFLY starts up with no node id

2. WFLY calculates a node id

3. WFLY checks JDBC store to see if this node id is known (under JDBC lock) and then writes node id to JDBC store to a table of known node ids and commits

4. WFLY writes node id to local configuration.xml

The issue is that there is no obvious way to garbage collect the table of known node ids for nodes that die (after step 3) and never come back.

Also, from a usability point of view if the user reverts to a config that doesn't have a node ID in the config those TX would be orphaned.