Wildlfy 15 Kubernetes cluster - KUBE_PING discovery protocol for JGroups cluster nodes managed by Kubernetes
alainwildfly Apr 11, 2019 11:03 AMHello,
I try to create a cluster of Wildlfy 15 nodes managed by Kubernetes.
I launch my instances with standalone-ha.xml file and use KUBE_PING as the discovery protocol for Jgroups cluster nodes.
I followed the official documentation https://docs.wildfly.org/15/High_Availability_Guide.html#discovery-for-kubernetes and jgroups-kubernetes/README.adoc at master · jgroups-extras/jgroups-kubernetes · GitHub
I also followed this tutorial but it was for Wildlfy 10: https://medium.com/@andrevcf/wildfly-10-kubernetes-cluster-dee7d4d377c6
For some reasons, the nodes do not communicate together.
I do not understand which configuration part is wrong.
Here is more details about the configuration:
1/Docker Image
My docker image is based on jboss/wildfly:15.0.1.Final
During the creation of the Docker image, I execute the following cli file script to disable default MPING protocol and enable KUBE_PING with the namespace set to ${KUBERNETES_NAMESPACE:kubeping-demo} and no label:
/subsystem=jgroups/stack=tcp/protocol=MPING:remove()
/subsystem=jgroups/stack=tcp/protocol=kubernetes.KUBE_PING:add(add-index=0,properties={namespace="${KUBERNETES_NAMESPACE:kubeping-demo}"})
This the corresponding definition of jboss:domain:jgroups subsytem in standalone-ha.xml file:
<subsystem xmlns="urn:jboss:domain:jgroups:6.0">
<channels default="ee">
<channel name="ee" stack="udp" cluster="ejb"/>
</channels>
<stacks>
...
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<protocol type="kubernetes.KUBE_PING">
<property name="namespace">
${KUBERNETES_NAMESPACE:kubeping-demo}
</property>
</protocol>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
</stacks>
</subsystem>
You will find the complete standalone-ha.xml file attached to this ticket.
2/KUBERNETES
This is the deployment.yaml file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: wildfly15-kubeping
namespace: kubeping-demo
spec:
selector:
matchLabels:
app: wildfly15-kubeping
replicas: 2
template:
metadata:
labels:
app: wildfly15-kubeping
spec:
serviceAccountName: jgroups-kubeping-service-account
containers:
- name: kube-ping
image: 172.31.21.69:5002/kube-ping:v10
command: ["/opt/jboss/wildfly/bin/standalone.sh"]
args: ["-c", "standalone-ha.xml", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0"]
resources:
requests:
memory: 300Mi
limits:
memory: 800Mi
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
- containerPort: 9990
- containerPort: 7600 #jgroups default
- containerPort: 8888 #jgroups undertow
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: KUBERNETES_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
imagePullSecrets:
- name: regcrednexus
3/Server logs
See logs.txt in attachment.
Kubernetes launched 2 nodes but in the logs of one of the node we see only one instance...
Received new cluster view for channel ejb: [wildfly15-kubeping-76765d5cc5-42tkv|0] (1) [wildfly15-kubeping-76765d5cc5-42tkv]
23:56:13,953 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Channel ejb local address is wildfly15-kubeping-76765d5cc5-42tkv, physical addresses are [127.0.0.1:55200]
23:56:13,953 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000079: Channel ejb local address is wildfly15-kubeping-76765d5cc5-42tkv, physical addresses are [127.0.0.1:55200]
23:56:13,998 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000078: Starting JGroups channel ejb
Thank you in advance for your help.
Alain
-
standalone-ha.xml.zip 5.9 KB
-
deployment.yaml.zip 670 bytes
-
kube_ping.cli.zip 304 bytes
-
logs.txt.zip 861 bytes