Wildlfy 15 Kubernetes cluster - KUBE_PING discovery protocol for JGroups cluster nodes managed by Kubernetes

[alainwildfly]

Hello,

I try to create a cluster of Wildlfy 15 nodes managed by Kubernetes.

I launch my instances with standalone-ha.xml file and use KUBE_PING as the discovery protocol for Jgroups cluster nodes.

I followed the official documentation https://docs.wildfly.org/15/High_Availability_Guide.html#discovery-for-kubernetes and jgroups-kubernetes/README.adoc at master · jgroups-extras/jgroups-kubernetes · GitHub

I also followed this tutorial but it was for Wildlfy 10: https://medium.com/@andrevcf/wildfly-10-kubernetes-cluster-dee7d4d377c6

For some reasons, the nodes do not communicate together.

I do not understand which configuration part is wrong.

Here is more details about the configuration:

1/Docker Image

My docker image is based on jboss/wildfly:15.0.1.Final

During the creation of the Docker image, I execute the following cli file script to disable default MPING protocol and enable KUBE_PING with the namespace set to ${KUBERNETES_NAMESPACE:kubeping-demo} and no label:

/subsystem=jgroups/stack=tcp/protocol=MPING:remove()

/subsystem=jgroups/stack=tcp/protocol=kubernetes.KUBE_PING:add(add-index=0,properties={namespace="${KUBERNETES_NAMESPACE:kubeping-demo}"})

This the corresponding definition of jboss:domain:jgroups subsytem in standalone-ha.xml file:

<subsystem xmlns="urn:jboss:domain:jgroups:6.0">

            <channels default="ee">

                <channel name="ee" stack="udp" cluster="ejb"/>

            </channels>

            <stacks>

     ...

                <stack name="tcp">

                    <transport type="TCP" socket-binding="jgroups-tcp"/>

                    <protocol type="kubernetes.KUBE_PING">

                        <property name="namespace">

                            ${KUBERNETES_NAMESPACE:kubeping-demo}

                        </property>

                    </protocol>

                    <protocol type="MERGE3"/>

                    <protocol type="FD_SOCK"/>

                    <protocol type="FD_ALL"/>

                    <protocol type="VERIFY_SUSPECT"/>

                    <protocol type="pbcast.NAKACK2"/>

                    <protocol type="UNICAST3"/>

                    <protocol type="pbcast.STABLE"/>

                    <protocol type="pbcast.GMS"/>

                    <protocol type="MFC"/>

                    <protocol type="FRAG3"/>

                </stack>

            </stacks>

        </subsystem>

You will find the complete standalone-ha.xml file attached to this ticket.

2/KUBERNETES

This is the deployment.yaml file:

apiVersion: apps/v1

kind: Deployment

metadata:

  name: wildfly15-kubeping

  namespace: kubeping-demo

spec:

  selector:

    matchLabels:

      app: wildfly15-kubeping

  replicas: 2

  template:

    metadata:

      labels:

        app: wildfly15-kubeping

    spec:

      serviceAccountName: jgroups-kubeping-service-account

      containers:

        - name: kube-ping

          image: 172.31.21.69:5002/kube-ping:v10

          command: ["/opt/jboss/wildfly/bin/standalone.sh"]

          args: ["-c", "standalone-ha.xml", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0"]

          resources:

requests:

memory: 300Mi

limits:

memory: 800Mi

          imagePullPolicy: IfNotPresent

          ports:

            - containerPort: 8080

            - containerPort: 9990

            - containerPort: 7600 #jgroups default

            - containerPort: 8888 #jgroups undertow

          env:

            - name: POD_NAME

valueFrom:

fieldRef:

apiVersion: v1

fieldPath: metadata.name

            - name: KUBERNETES_NAMESPACE

valueFrom:

fieldRef:

apiVersion: v1

fieldPath: metadata.namespace

            - name: POD_IP

valueFrom:

fieldRef:

apiVersion: v1

fieldPath: status.podIP

      imagePullSecrets:

      - name: regcrednexus

3/Server logs

See logs.txt in attachment.

Kubernetes launched 2 nodes but in the logs of one of the node we see only one instance...

Received new cluster view for channel ejb: [wildfly15-kubeping-76765d5cc5-42tkv|0] (1) [wildfly15-kubeping-76765d5cc5-42tkv]

23:56:13,953 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Channel ejb local address is wildfly15-kubeping-76765d5cc5-42tkv, physical addresses are [127.0.0.1:55200]

23:56:13,953 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000079: Channel ejb local address is wildfly15-kubeping-76765d5cc5-42tkv, physical addresses are [127.0.0.1:55200]

23:56:13,998 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000078: Starting JGroups channel ejb

Thank you in advance for your help.

Alain

[alainwildfly]

Hello,

I addition to my previous message, I wanted to double-check that the information provided by the cli-file is sufficient.

With this cli file, we just activate the KUBE_PING TCP-Protocol and specify a namespace (as detailed in the official documentation)

We did not specify a bind_port nor a Port Range.

In the configuration file for KUBE_PING and jgroups  provided by Bela Ban, we have a lot of other parameters (see uploaded config.xml file)

Thank you again for your help.

Alain

[belaban]

Are you running your 2 nodes on minikube/minishift *locally*? The nodes bind to 127.0.0.1; this will only work if both pods are running on the same physical box, but not (for example) on GKE.

[alainwildfly]

Dear Bela Ban,

Thank you for your answer.

We are working on Amazon EKS. The 2 nodes a running on separate PODS on different machines.

In your message, when you say that the nodes bind to 127.0.0.1, is it because of what you have seen in the logs?

23:56:14,000 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-2) ISPN000094: Received new cluster view for channel ejb: [wildfly15-kubeping-76765d5cc5-42tkv|0] (1) [wildfly15-kubeping-76765d5cc5-42tkv]

23:56:14,000 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Channel ejb local address is wildfly15-kubeping-76765d5cc5-42tkv, physical addresses are [127.0.0.1:55200]

We do not set the value 127.0.0.1 explicitely. I do not know for which reason the real IP address is not resolved.

I am quite sure I forget something in the configuration.

In you example from jgroups-kubernetes/README.adoc at master · jgroups-extras/jgroups-kubernetes · GitHub , we have the following parameters

<TCP

   bind_addr="loopback,match-interface:eth0"

   bind_port="7800"

  ...

  />

<org.jgroups.protocols.kubernetes.KUBE_PING

   port_range="1"

   namespace="${KUBERNETES_NAMESPACE:production}"

   labels="${KUBERNETES_LABEL:cluster=nyc}"

  />

I do not specify the bin_addr nor the bind_port in the TCP tag for Wildlfy.

I just activated KUBE_PING and specified a namespace (no label):

/subsystem=jgroups/stack=tcp/protocol=MPING:remove()

/subsystem=jgroups/stack=tcp/protocol=kubernetes.KUBE_PING:add(add-index=0,properties={namespace="${KUBERNETES_NAMESPACE:kubeping-demo}"})

Should I also define those TCP parameters?

What else should I add in the configuration to be able to work with nodes deployed on different  physical boxes?

Thank you a lot in advance for your help because I am really blocked with this issue.

Kind regards,

Alain Morlet