Changing node-id with reload required in WildFly during boot time

[ochaloup]

Hi all,

I would like to ask for feedback about the way of changing Narayana node-id in WFLY. Recently there was created an issue regarding this

[WFLY-12317] Using JTA transaction's node_name attribute is set to an old value after node-identifier is changed.

The reason is that the model node defines the changing of the node-id is fine with :reload operation - WildFly 16.0 Model Reference

That was working fine up to the time I did changes for timeout processing for issue [WFLY-11811] Default transaction timeout could be wrongly configured because of the TM initiation race condition. The initialization order was changed during WFLY boot time and the node-id is needed with restart JVM (wildfly/TransactionSubsystemRootResourceDefinition.java at master · wildfly/wildfly · GitHub ).

The question is if the reload required for the node-id should be preserved or the JVM restart should be required from now.

For the option of the reload required the would be to access `TxControl` and define the nodeIdentifier during the transaction service star - that's called during restart (like this [WFLY-12317] Using JTA transaction's node_name attribute is set to an old value after node-identifier is changed).

As Mike refers in his comment [WFLY-12317] Using JTA transaction's node_name attribute is set to an old value after node-identifier is changed, the safety should be considered.

The CoreEnvironmentService already defines the node-id during startup (wildfly/CoreEnvironmentService.java at 17.0.1.Final · wildfly/wildfly · GitHub ). But the code does not approach directly to the TxControl class. It sounds that the reload-required for the node-id was intentional.

Would there be some ideas on this?

Thanks

Ondra

[tomjenkinson]

I do note some precedence for editing TxControl wildfly/ArjunaTransactionManagerService.java at d402d02c54f26cf66ecdd462354333d2323b01d7 · wildfly/wildfly · GitHub  wildfly/TransactionSubsystemRootResourceDefinition.java at 226a2ef4aeb473f4d95cad3487296f6191cd5096 · wildfly/wildfly · …

I think the main thing to check would be to make sure that everything is shutdown first and if there is anything resident in memory it shouldn't have a reference to the node ID. Maybe others have some further suggestions?

[ochaloup]

Sure, the TxControl was necessary for the timeout would be changeable without the need of the JVM restart - see our discussion at [WFLY-11811] explicitly passing txn default timeout to TxControl during server startup by ochaloup · Pull Request #12140…

The change caused the JVM static properties being loaded in a different order than before.

Still holds true that the CoreEnvironmentService#start tried to change the node-id without taking into account that TxControl is statically defined. The change to the CoreEnvironmentalBean has no effect (wildfly/CoreEnvironmentService.java at 17.0.1.Final) after reload.

In fact, has no effect after reload even before the reffered change (wildfly/ArjunaTransactionManagerService.java at d402d02c54f26cf66ecdd462354333d2323b01d7). If the node identifier was defined and e.g. recovery was run (something that touched the TxControl) and then somebody tries to change node-id with only reload (as the cli shows)

/subsystem=transactions:write-attribute(name=node-identifier, value=newnodeid)
{
    "outcome" => "success",
    "response-headers" => {
        "operation-requires-reload" => true,
        "process-state" => "reload-required"
    }
}

after reload the old node id was left for the recovery processing. The cli shows the id was changed but the TxControl already loaded the static variable.

Few notes from my investigation

What I'm looking to the code and running debugger the call for the reload and shutdown is the same in the method call outcome but different in the way how they are called. Service is called for the stop and suspend for both reload and shutdown. Shutdown contains the timeout parameter which defines time to wait for finishing all requests before the server is stopped. Reload does not have such functionality. It just stops the connections and start the service again.

All requests are stopped - for example the in-flight http requests. Thus transactions related to them are stopped/rolled-back. But this is disputable.

Every subsystem handles the stopping of the request on its own and I'm not fully sure about eg. ejb, jca inbound stuff, batch processing etc.

[ochaloup]

After the discussion in the team it was agreed that the Narayana codebase contains static properties for configuration which are brought as a bit burden from history. The refactoring of that approach requires a deep investigation on the code. The JVM restart is the correct resolution  for current situation as it ensures safety and data consistency.