Hi dsargrad ,

there is unfortunately no overall documentation on this topic. But we've been recently working on integration of Narayana in WildFly with Kubernetes. So you may find some related information here:

wildfly/EJB_on_Kubernetes.adoc at master · wildfly/wildfly · GitHub

wildfly-operator/user-guide.adoc at master · wildfly/wildfly-operator · GitHub

When getting back to your question. It would be good to know what's your expectation (and architecture of your application). As you posted the example of the Narayana and Spring boot integration, do you run with Spring boot? Do you use some clustering solution over nodes/pods (or only one node is used)? Do you have e.g. some loadbalancing in front of your app (then you probably need a stickysession with the requests to Narayana node)?

On top of that - as you are asking for transaction manager I expect you use XA transactions, right? Why I'm asking it's summarized in a related discussion here: Re: Quarkus in Google App Engine.

What is needed to understand is that Narayana, in its current state of development, is "a stateful" component. Progress of XA transaction is needed to be persisted somewhere. Narayana has to be able to access that persistent storage to finish with the XA transaction and the storage has to be available to the Narayana even after the container/pod restarts.

Narayana is not capable to share the storage state over multiple nodes/pods. Every pod has it's own place where data is stored. You may configure Narayana to use either filesystem or JDBC storage.

If you use the file system then probably the best way to go is to use the StatefulSet. When node is started you claim the PV which is bound to the particular pod. When the pod crashes, the pod is relocated or the app is restarted with a new version then the partition is still available to the particular pod. Narayana is then capable to access the data and finish all unfinished XA transactions even after restart.

If you use the JDBC storage then you may use the StatefulSet as well. What is important is that the pod should be bound to the particular connection - database - database table. After the pod is restarted it still needs to be able to access the same storage.

The second important point from the perspective of XA transaction recovery in Narayana is that the configuration needs to specify unique node identifier. That identifier needs to be bound the same for the Narayana instance at the pod and the storage that's used.

Depending what runtime you use the node identifier could be configured differently. For standalone Narayana the configuration is done with xml descriptor (e.g. here narayana/jbossts-properties.xml at master · jbosstm/narayana · GitHub) or with system properties of the same name.

From what was said the StatefulSet is, I think, currently the best option how to configure Narayana on Kubernetes. If you use the file system, then you should mount the Narayana object store to the StatefulSet PV and the node identifier should be defined as name of the StatefulSet pod.

On top of that you need to take care when you scale down the StatefulSet (you decreases the number of pods). In such case you need first check that Narayana finished all the transactions (the object store is clean) and then you can remove the pod from the StatefulSet. Otherwise you risk the transaction data inconsistencies.

Ondra

Thanks for the very comprehensive response Ondra!

I would like to emphasize two points:

1. Our work to date has been to get this tested as working on OpenShift, not to test this on raw Kubernetes as yet

2. The node id must be unique and must be used in combination with the object store used for any previous execution of the transaction manager with this node id, this is of fundamental importance for data integrity