3 Replies Latest reply on Apr 16, 2019 4:07 PM by Alain Morlet

    Wildlfy 15 Kubernetes cluster - KUBE_PING discovery protocol for JGroups cluster nodes managed by Kubernetes

    Alain Morlet Newbie

      Hello,

      I try to create a cluster of Wildlfy 15 nodes managed by Kubernetes.

      I launch my instances with standalone-ha.xml file and use KUBE_PING as the discovery protocol for Jgroups cluster nodes.

       

      I followed the official documentation https://docs.wildfly.org/15/High_Availability_Guide.html#discovery-for-kubernetes and jgroups-kubernetes/README.adoc at master · jgroups-extras/jgroups-kubernetes · GitHub

      I also followed this tutorial but it was for Wildlfy 10: https://medium.com/@andrevcf/wildfly-10-kubernetes-cluster-dee7d4d377c6

       

      For some reasons, the nodes do not communicate together.

      I do not understand which configuration part is wrong.

       

      Here is more details about the configuration:

       

      1/Docker Image

      My docker image is based on jboss/wildfly:15.0.1.Final

       

      During the creation of the Docker image, I execute the following cli file script to disable default MPING protocol and enable KUBE_PING with the namespace set to ${KUBERNETES_NAMESPACE:kubeping-demo} and no label:

      /subsystem=jgroups/stack=tcp/protocol=MPING:remove()

      /subsystem=jgroups/stack=tcp/protocol=kubernetes.KUBE_PING:add(add-index=0,properties={namespace="${KUBERNETES_NAMESPACE:kubeping-demo}"})

       

      This the corresponding definition of jboss:domain:jgroups subsytem in standalone-ha.xml file:

       

      <subsystem xmlns="urn:jboss:domain:jgroups:6.0">

                  <channels default="ee">

                      <channel name="ee" stack="udp" cluster="ejb"/>

                  </channels>

                  <stacks>

           ...

                      <stack name="tcp">

                          <transport type="TCP" socket-binding="jgroups-tcp"/>

                          <protocol type="kubernetes.KUBE_PING">

                              <property name="namespace">

                                  ${KUBERNETES_NAMESPACE:kubeping-demo}

                              </property>

                          </protocol>

                          <protocol type="MERGE3"/>

                          <protocol type="FD_SOCK"/>

                          <protocol type="FD_ALL"/>

                          <protocol type="VERIFY_SUSPECT"/>

                          <protocol type="pbcast.NAKACK2"/>

                          <protocol type="UNICAST3"/>

                          <protocol type="pbcast.STABLE"/>

                          <protocol type="pbcast.GMS"/>

                          <protocol type="MFC"/>

                          <protocol type="FRAG3"/>

                      </stack>

                  </stacks>

              </subsystem>

       

      You will find the complete standalone-ha.xml file attached to this ticket.

       

      2/KUBERNETES

      This is the deployment.yaml file:

       

      apiVersion: apps/v1

      kind: Deployment

      metadata:

        name: wildfly15-kubeping

        namespace: kubeping-demo

      spec:

        selector:

          matchLabels:

            app: wildfly15-kubeping

        replicas: 2

        template:

          metadata:

            labels:

              app: wildfly15-kubeping

          spec:

            serviceAccountName: jgroups-kubeping-service-account

            containers:

              - name: kube-ping

                image: 172.31.21.69:5002/kube-ping:v10

                command: ["/opt/jboss/wildfly/bin/standalone.sh"]

                args: ["-c", "standalone-ha.xml", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0"]

                resources:

      requests:

      memory: 300Mi

      limits:

      memory: 800Mi

                imagePullPolicy: IfNotPresent

                ports:

                  - containerPort: 8080

                  - containerPort: 9990

                  - containerPort: 7600 #jgroups default

                  - containerPort: 8888 #jgroups undertow

                env:

                  - name: POD_NAME

      valueFrom:

      fieldRef:

      apiVersion: v1

      fieldPath: metadata.name

                  - name: KUBERNETES_NAMESPACE

      valueFrom:

      fieldRef:

      apiVersion: v1

      fieldPath: metadata.namespace

                  - name: POD_IP

      valueFrom:

      fieldRef:

      apiVersion: v1

      fieldPath: status.podIP

            imagePullSecrets:

            - name: regcrednexus

       

      3/Server logs

      See logs.txt in attachment.

       

      Kubernetes launched 2 nodes but in the logs of one of the node we see only one instance...

       

      Received new cluster view for channel ejb: [wildfly15-kubeping-76765d5cc5-42tkv|0] (1) [wildfly15-kubeping-76765d5cc5-42tkv]

      23:56:13,953 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Channel ejb local address is wildfly15-kubeping-76765d5cc5-42tkv, physical addresses are [127.0.0.1:55200]

      23:56:13,953 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000079: Channel ejb local address is wildfly15-kubeping-76765d5cc5-42tkv, physical addresses are [127.0.0.1:55200]

      23:56:13,998 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000078: Starting JGroups channel ejb

       

       

      Thank you in advance for your help.

      Alain

        • 1. Re: Wildlfy 15 Kubernetes cluster - KUBE_PING discovery protocol for JGroups cluster nodes managed by Kubernetes
          Alain Morlet Newbie

          Hello,

          I addition to my previous message, I wanted to double-check that the information provided by the cli-file is sufficient.

          With this cli file, we just activate the KUBE_PING TCP-Protocol and specify a namespace (as detailed in the official documentation)

          We did not specify a bind_port nor a Port Range.

           

          In the configuration file for KUBE_PING and jgroups  provided by Bela Ban, we have a lot of other parameters (see uploaded config.xml file)

           

          Thank you again for your help.

           

          Alain

          • 2. Re: Wildlfy 15 Kubernetes cluster - KUBE_PING discovery protocol for JGroups cluster nodes managed by Kubernetes
            Bela Ban Master

            Are you running your 2 nodes on minikube/minishift *locally*? The nodes bind to 127.0.0.1; this will only work if both pods are running on the same physical box, but not (for example) on GKE.

            • 3. Re: Wildlfy 15 Kubernetes cluster - KUBE_PING discovery protocol for JGroups cluster nodes managed by Kubernetes
              Alain Morlet Newbie

              Dear Bela Ban,

              Thank you for your answer.

              We are working on Amazon EKS. The 2 nodes a running on separate PODS on different machines.

               

              In your message, when you say that the nodes bind to 127.0.0.1, is it because of what you have seen in the logs?

              23:56:14,000 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-2) ISPN000094: Received new cluster view for channel ejb: [wildfly15-kubeping-76765d5cc5-42tkv|0] (1) [wildfly15-kubeping-76765d5cc5-42tkv]

              23:56:14,000 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Channel ejb local address is wildfly15-kubeping-76765d5cc5-42tkv, physical addresses are [127.0.0.1:55200]

               

              We do not set the value 127.0.0.1 explicitely. I do not know for which reason the real IP address is not resolved.

              I am quite sure I forget something in the configuration.

               

              In you example from jgroups-kubernetes/README.adoc at master · jgroups-extras/jgroups-kubernetes · GitHub , we have the following parameters

              <TCP

                 bind_addr="loopback,match-interface:eth0"

                 bind_port="7800"

                ...

                />

              <org.jgroups.protocols.kubernetes.KUBE_PING

                 port_range="1"

                 namespace="${KUBERNETES_NAMESPACE:production}"

                 labels="${KUBERNETES_LABEL:cluster=nyc}"

                />

               

              I do not specify the bin_addr nor the bind_port in the TCP tag for Wildlfy.

               

              I just activated KUBE_PING and specified a namespace (no label):

              /subsystem=jgroups/stack=tcp/protocol=MPING:remove()

              /subsystem=jgroups/stack=tcp/protocol=kubernetes.KUBE_PING:add(add-index=0,properties={namespace="${KUBERNETES_NAMESPACE:kubeping-demo}"})

               

              Should I also define those TCP parameters?

              What else should I add in the configuration to be able to work with nodes deployed on different  physical boxes?

               

              Thank you a lot in advance for your help because I am really blocked with this issue.

               

              Kind regards,

              Alain Morlet