1 2 Previous Next 19 Replies Latest reply on Apr 9, 2019 5:55 AM by cablespaghetti

WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

thradec Mar 5, 2019 6:03 AM

Hi,

I'm trying to run keycloak 4.8.3 in kubernetes cluser and I noticed following warning in logs

WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

basically, I'm using a slightly modified helm chart charts/stable/keycloak at master · helm/charts · GitHub which use DNS_PING discovery protocol, here is the snippet from standalone-ha.xml

        <subsystem xmlns="urn:jboss:domain:jgroups:6.0">
            <channels default="ee">
                <channel name="ee" stack="tcp" cluster="ejb"/>
            </channels>
            <stacks>
                <stack name="tcp">
                    <transport type="TCP" socket-binding="jgroups-tcp"/>
                    <protocol type="dns.DNS_PING">
                        <property name="dns_query">
                            keycloak-headless.ph00a1-sso.svc.cluster.local
                        </property>
                    </protocol>
                    <protocol type="MERGE3"/>
                    <protocol type="FD_SOCK"/>
                    <protocol type="FD_ALL"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK2"/>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="pbcast.GMS"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG3"/>
                </stack>

The warning in the logs appears when some pod is deleted and new one is started, eg. during rolling new version. Before that, everything looks fine and the cluster is formed propely (at least it seems from logs, eg.)

 [0m [0m13:31:01,512 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000078: Starting JGroups channel ejb
 [0m [0m13:31:01,512 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000078: Starting JGroups channel ejb
 [0m [0m13:31:01,519 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|1] (2) [keycloak-0, keycloak-1]
 [0m [0m13:31:01,519 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-2) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|1] (2) [keycloak-0, keycloak-1]
 [0m [0m13:31:01,612 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000079: Channel ejb local address is keycloak-1, physical addresses are [10.236.68.237:7600]
 [0m [0m13:31:01,612 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Channel ejb local address is keycloak-1, physical addresses are [10.236.68.237:7600]
 [0m [0m13:31:01,720 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-2) ISPN000128: Infinispan version: Infinispan 'Estrella Galicia' 9.3.1.Final

The only similar issue which I found is [ISPN-7489] org.jgroups.protocols.TCP emits errors when node leaves the cluster - JBoss Issue Tracker , but it should be fixed already.

I'd be glad for any advice.

Thanks a lot

Tomas

1. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

belaban Mar 8, 2019 3:25 AM (in response to thradec)

When you're execing into one of the pods, what's the output of 'dig keycloak-headless.ph00a1-sso.svc.cluster.local'?
Are you running this on minikube/minishift? In my own testing, I found that minikube's DNS service sometimes delays DNS responses. I'm not too familiar with minikube, but it seemed to be running 2 DNS services, instead of 1.
Please also want to experiment with setting dns_address, see [1] for reference.
[1] Reliable group communication with JGroups
Actions

2. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

thradec Mar 8, 2019 7:47 AM (in response to belaban)

Hi and thanks for the reply

this is the output from dig cmd

[jboss@keycloak-0 ~]$ dig keycloak-headless.ph00a1-sso.svc.cluster.local


; <<>> DiG 9.9.4-RedHat-9.9.4-73.el7_6 <<>> keycloak-headless.ph00a1-sso.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12215
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1


;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;keycloak-headless.ph00a1-sso.svc.cluster.local. IN A


;; ANSWER SECTION:
keycloak-headless.ph00a1-sso.svc.cluster.local. 1 IN A 10.236.68.37
keycloak-headless.ph00a1-sso.svc.cluster.local. 1 IN A 10.236.66.133
keycloak-headless.ph00a1-sso.svc.cluster.local. 1 IN A 10.236.69.182


;; Query time: 0 msec
;; SERVER: 10.236.0.10#53(10.236.0.10)
;; WHEN: Fri Mar 08 12:38:22 UTC 2019
;; MSG SIZE  rcvd: 123

the application is running in cluster build by RKE Rancher Kubernetes Engine, the following is the list and version of used components

  kubernetes:  rancher-hyperkube:v1.12.3-rancher1
  etcd:  rancher-coreos-etcd:v3.2.24
  alpine:  rancher-rke-tools:v0.1.16
  nginx_proxy:  rancher-rke-tools:v0.1.16
  cert_downloader:  rancher-rke-tools:v0.1.16
  kubernetes_services_sidecar:  rancher-rke-tools:v0.1.16
  kubedns:  rancher-k8s-dns-kube-dns-amd64:1.14.13
  dnsmasq:  rancher-k8s-dns-dnsmasq-nanny-amd64:1.14.13
  kubedns_sidecar:  rancher-k8s-dns-sidecar-amd64:1.14.13
  kubedns_autoscaler:  rancher-cluster-proportional-autoscaler-amd64:1.0.0
  pod_infra_container:  rancher-pause-amd64:3.1

3. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

sebastian.laskawiec Mar 8, 2019 8:37 AM (in response to thradec)
So it seems Kubernetes killed a Pod and JGroups didn't noticed that it already disappeared.

If you're running Kubernetes Rolling Update with more than 2 nodes (I see you're running 3), you should be good and this should do no harm to you.

There are two things that can cause this that come into my mind:
FD_* protocols didn't notice the change. The killed pod should be already removed from the view.
logical_addr_cache_expiration is still too high in the TP.

belaban What do you think about this?

It would be great if you could help us diagnosing this Tomas. Unfortunately, both require changes in the xml configuration.

The latter is much easier to verify and I highly recommend to start with that. Here's a configuration snippet:
<transport type="TCP" socket-binding="jgroups-tcp" >

<property name="logical_addr_cache_expiration">260000</property>
</transport>

As for the former, you may take a look at one of my old configuration tuning guides: presentations/README.asciidoc at master · slaskawi/presentations · GitHub
In essence, you'd need to remove VERIFY_SUSPECT and tune FD_ALL settings. You may start with copy-pasting things I have there.
Actions
4. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

thradec Mar 8, 2019 3:18 PM (in response to sebastian.laskawiec)

Hi, thanks for the hints, I'd be glad to help you find the issue, I'll try to change the xml configuration and let you know if it helps
Actions
5. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

thradec Mar 11, 2019 1:06 PM (in response to sebastian.laskawiec)

Hi, I tried to set logical_addr_cache_expiration property, but the warning is still there, I'll try the others settings, which you mentioned.
Actions
6. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

cablespaghetti Mar 13, 2019 1:52 PM (in response to thradec)
We have this exact same issue using that same Helm chart (v4.4.0) with Keycloak 4.8.3. It does seem to have appeared after we restarted all the containers. Any progress with working out the fix?

edit: I've now added this to my Helm values file, to disable Java's idiotic infinite DNS caching. This seems to have fixed the problem for me although only time will tell...
## Additional arguments to start command e.g. -Dkeycloak.import= to load a realm extraArgs: "-Dsun.net.inetaddr.ttl=0"
Actions
7. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

sebastian.laskawiec Mar 14, 2019 4:48 AM (in response to cablespaghetti)

Unfortunately no progress Sam, Tomas. I tried to reproduce this locally a number of times and it seems this has something to do with your environment.

Sam's idea about lowering TTL on DNS cache might work. As I said before, the scenario is that a node disappears (Kubernetes kills it) and JGroups don't notice that on time.

I also repeat what I said before - unless you have a really weird setup, this will do no harm. JGroups should update the view shortly after this warning is emitted.
Actions
8. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

cablespaghetti Mar 14, 2019 6:27 AM (in response to sebastian.laskawiec)

Unfortunately my "fix" stopped working again after I left work yesterday. The error keeps printing every minute or so forever, so it's not a temporary thing until JGroups updates.
Actions
9. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

thradec Mar 14, 2019 6:42 AM (in response to sebastian.laskawiec)

Hi, neither I have any positive progress.

- I tried some other configuration changes (eg. remove VERIFY_SUSPECT and set timeouts in FD_ALL), but the cluster start to be even more unstable, so I had to revert
- I can confirm the same behavior as Sam, that the warning is permanent and doesn't disappear after a while
- but luckily I didn't observer any negative influence on functionality

Can I somehow check the current view of JGroups?
Actions
10. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

cablespaghetti Mar 14, 2019 9:01 AM (in response to thradec)

Tomas. Are you running on AWS by any chance? We're using EKS with the AWS VPC CNI Plugin. I notice that JGroups uses multicast for some of its functionality...I wonder if Amazon's lack of multicast support is causing this...
Actions

11. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

tdarimont Mar 14, 2019 6:44 PM (in response to thradec)

Hello,

I see this problem too in a 3 node Keycloak cluster, which I created with the Keycloak Helm Chart running on Azure AKS. As test I deleted the keycloak-2 pod, which is then automatically recreated.

The cluster seems to behave normally, but one can see the mentioned warn messages in the log:


keycloak-1 keycloak 17:22:48,312 INFO  [org.infinispan.CLUSTER] (remote-thread--p17-t1) [Context=client-mappings] ISPN100002: Starting rebalance with members [keycloak-1, keycloak-0, keycloak-2], phase READ_OLD_WRITE_ALL, topology id 27
keycloak-1 keycloak 17:22:49,814 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=work] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 28
keycloak-1 keycloak 17:22:50,090 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=work] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 29
keycloak-1 keycloak 17:22:50,101 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=sessions] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 47
keycloak-1 keycloak 17:22:50,103 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=offlineSessions] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 47
keycloak-1 keycloak 17:22:50,104 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=actionTokens] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 48
keycloak-1 keycloak 17:22:50,106 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=clientSessions] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 46
keycloak-0 keycloak 18:46:10,946 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:46:45,085 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:47:19,978 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:47:48,271 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:48:17,167 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:49:00,858 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:49:40,540 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:22:50,391 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t23) [Context=clientSessions] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 47
keycloak-1 keycloak 17:22:50,391 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t22) [Context=actionTokens] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 49
keycloak-1 keycloak 17:22:50,391 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t30) [Context=sessions] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 48
keycloak-1 keycloak 17:22:50,392 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=work] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 30
keycloak-1 keycloak 17:22:50,392 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t28) [Context=offlineSessions] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 48
keycloak-0 keycloak 18:50:18,760 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:50:55,692 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:51:34,666 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:22:50,394 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=offlineClientSessions] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 49
keycloak-1 keycloak 17:22:50,395 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=authenticationSessions] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 48
keycloak-0 keycloak 18:52:00,242 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:52:18,624 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:52:40,562 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak-1 keycloak 18:52:58,724 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak 17:22:50,399 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t28) [Context=loginFailures] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 48
keycloak-1 keycloak 17:22:50,692 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=sessions] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 49
keycloak-1 keycloak 17:22:50,693 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t22) [Context=offlineClientSessions] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 50
keycloak-1 keycloak 17:22:50,693 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t25) [Context=authenticationSessions] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 49
keycloak-1 keycloak 17:22:50,693 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t23) [Context=clientSessions] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 48
keycloak-1 keycloak 17:22:50,692 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t30) [Context=offlineSessions] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 49
keycloak-1 keycloak 17:22:50,692 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t26) [Context=loginFailures] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 49
keycloak-1 keycloak 17:22:50,697 INFO  [org.infinispan.CLUSTER] (remote-thread--p17-t1) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 28
keycloak-1 keycloak 17:22:50,696 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t28) [Context=actionTokens] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 50
keycloak-1 keycloak 17:22:50,993 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t26) [Context=offlineClientSessions] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 51
keycloak-1 keycloak 17:22:50,993 INFO  [org.infinispan.CLUSTER] (remote-thread--p17-t1) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 29
keycloak-1 keycloak 17:22:50,993 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t28) [Context=authenticationSessions] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 50
keycloak-1 keycloak 17:22:50,993 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t30) [Context=loginFailures] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 50
keycloak-0 keycloak 18:53:23,710 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:53:40,948 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:54:17,090 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:55:01,945 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:55:28,188 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:56:01,495 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:56:43,653 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:56:59,333 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:57:24,373 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:57:49,312 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:58:09,373 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:58:52,386 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 18:59:35,883 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:00:13,669 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:00:55,143 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:01:17,637 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:01:53,316 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:02:21,274 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:02:45,114 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:03:19,536 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:03:35,989 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:03:51,034 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:22:51,294 INFO  [org.infinispan.CLUSTER] (remote-thread--p17-t1) [Context=client-mappings] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 30
keycloak-1 keycloak 17:28:58,859 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloakkeycloak-0  17:29:35,708 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:30:00,904 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:30:45,616 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:31:10,447 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:31:48,612 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:32:19,204 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak keycloak 17:32:43,272 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:33:06,470 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:33:23,396 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 19:04:09,147 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:04:50,647 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:05:05,835 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:05:29,407 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:06:12,327 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak-0 keycloak 19:06:54,637 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
keycloak 17:34:01,096 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:34:26,468 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:34:45,733 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:35:11,125 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:35:51,240 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:36:07,178 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:36:28,214 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:37:06,967 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:37:22,194 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:37:57,296 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:38:41,005 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:39:11,305 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:39:30,659 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:40:12,674 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:40:51,718 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:41:06,955 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:41:43,394 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:42:26,488 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:43:00,361 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:43:25,884 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:43:53,646 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:44:08,871 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:44:41,504 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:45:09,314 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:45:53,227 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:46:21,455 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:47:00,441 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:47:32,014 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:48:10,429 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:48:34,421 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:49:08,532 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:49:50,324 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:50:20,557 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:50:51,530 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:51:34,304 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
keycloak-1 keycloak 17:52:09,189 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message

The JGroups config looks like this:

        <subsystem xmlns="urn:jboss:domain:jgroups:6.0">
            <channels default="ee">
                <channel name="ee" stack="tcp" cluster="ejb"/>
            </channels>
            <stacks>
                <stack name="udp">
                    <transport type="UDP" socket-binding="jgroups-udp"/>
                    <protocol type="dns.DNS_PING">
                        <property name="dns_query">
                            keycloak-headless.default.svc.cluster.local
                        </property>
                    </protocol>
                    <protocol type="MERGE3"/>
                    <protocol type="FD_SOCK"/>
                    <protocol type="FD_ALL"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK2"/>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="pbcast.GMS"/>
                    <protocol type="UFC"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG3"/>
                </stack>
                <stack name="tcp">
                    <transport type="TCP" socket-binding="jgroups-tcp"/>
                    <protocol type="dns.DNS_PING">
                        <property name="dns_query">
                            keycloak-headless.default.svc.cluster.local
                        </property>
                    </protocol>
                    <protocol type="MERGE3"/>
                    <protocol type="FD_SOCK"/>
                    <protocol type="FD_ALL"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK2"/>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="pbcast.GMS"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG3"/>
                </stack>
            </stacks>
        </subsystem>

Cheers,

Thomas

12. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

tdarimont Mar 14, 2019 6:46 PM (in response to tdarimont)

Quick update: I just tried dns.DNS_PING with Keycloak 4.8.3.Final and the latest jgroups 4.0.19 (which was just released today ;-) ) on my 3-node Kubernetes Cluster running on Azure AKS.

(I added the new jgroups-4.0.19.Final.jar to the jgroups/main module and updated the module.xml to refer to the new jar).

With jgroups-4.0.19.Final.jar I don't see error messages like "no physical address for keycloak-2, dropping message" anymore

So it seems that upgrading the jgroups version helps to get rid of this problem - note, that I did not change the dns.DNS_PING configuration.

I saw that the issue [JGRP-2316] DNS_PING is not using correct ports with SRV based service discovery - JBoss Issue Tracker is already fixed with 4.0.16, so I guess one could also try that version.

# Killed Pod keycloak-2


keycloak-2 keycloak *** JBossAS process (228) received TERM signal ***
keycloak-0 keycloak 22:27:58,030 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t15) [Context=sessions] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 23
keycloak-1 keycloak 22:27:58,031 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|4] (2) [keycloak-0, keycloak-1]
keycloak-0 keycloak 22:27:58,034 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t11) [Context=loginFailures] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 22
keycloak-0 keycloak 22:27:58,038 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t13) [Context=offlineSessions] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 23
keycloak-1 keycloak 22:27:58,038 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN100001: Node keycloak-2 left the cluster
keycloak-1 keycloak 22:27:58,041 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|4] (2) [keycloak-0, keycloak-1]
keycloak-0 keycloak 22:27:58,039 INFO  [org.infinispan.CLUSTER] (remote-thread--p17-t3) [Context=client-mappings] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 23
keycloak-1 keycloak 22:27:58,043 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN100001: Node keycloak-2 left the cluster
keycloak-1 keycloak 22:27:58,044 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|4] (2) [keycloak-0, keycloak-1]
keycloak-1 keycloak 22:27:58,045 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN100001: Node keycloak-2 left the cluster
keycloak-1 keycloak 22:27:58,045 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|4] (2) [keycloak-0, keycloak-1]
keycloak-1 keycloak 22:27:58,045 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN100001: Node keycloak-2 left the cluster
keycloak-1 keycloak 22:27:58,046 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|4] (2) [keycloak-0, keycloak-1]
keycloak-1 keycloak 22:27:58,046 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN100001: Node keycloak-2 left the cluster
keycloak-0 keycloak 22:27:58,051 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t10) [Context=authenticationSessions] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 23
keycloak-0 keycloak 22:27:58,051 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t15) [Context=clientSessions] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 21
keycloak-0 keycloak 22:27:58,052 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t11) [Context=work] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 21
keycloak-0 keycloak 22:27:58,052 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t13) [Context=offlineClientSessions] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 24
keycloak-0 keycloak 22:27:58,055 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t13) [Context=actionTokens] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 21
- keycloak-2
+ keycloak-2 › keycloak


# Pod keycloak-2 restarts
...
keycloak-2 keycloak 22:28:28,219 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 39) WFLYCLJG0001: Activating JGroups subsystem. JGroups version 4.0.19
...
keycloak-0 keycloak 22:28:31,121 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
keycloak-0 keycloak 22:28:31,121 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN100000: Node keycloak-2 joined the cluster
keycloak-0 keycloak 22:28:31,122 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
keycloak-0 keycloak 22:28:31,122 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN100000: Node keycloak-2 joined the cluster
keycloak-0 keycloak 22:28:31,122 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
keycloak-0 keycloak 22:28:31,122 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN100000: Node keycloak-2 joined the cluster
keycloak-0 keycloak 22:28:31,130 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
keycloak-0 keycloak 22:28:31,134 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN100000: Node keycloak-2 joined the cluster
keycloak-0 keycloak 22:28:31,138 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
keycloak-0 keycloak 22:28:31,138 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN100000: Node keycloak-2 joined the cluster
keycloak-1 keycloak 22:28:31,147 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
keycloak-1 keycloak 22:28:31,148 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN100000: Node keycloak-2 joined the cluster
keycloak-1 keycloak 22:28:31,151 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
keycloak-1 keycloak 22:28:31,152 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN100000: Node keycloak-2 joined the cluster
keycloak-1 keycloak 22:28:31,152 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
keycloak-1 keycloak 22:28:31,153 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN100000: Node keycloak-2 joined the cluster
keycloak-1 keycloak 22:28:31,155 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
keycloak-1 keycloak 22:28:31,155 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN100000: Node keycloak-2 joined the cluster
keycloak-1 keycloak 22:28:31,156 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
keycloak-1 keycloak 22:28:31,156 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN100000: Node keycloak-2 joined the cluster
keycloak-2 keycloak 22:28:31,664 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000078: Starting JGroups channel ejb
keycloak-2 keycloak 22:28:31,667 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000078: Starting JGroups channel ejb
keycloak-2 keycloak 22:28:31,681 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-2) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
keycloak-2 keycloak 22:28:31,681 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
keycloak-2 keycloak 22:28:31,701 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000079: Channel ejb local address is keycloak-2, physical addresses are [10.244.0.27:7600]
keycloak-2 keycloak 22:28:31,713 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Channel ejb local address is keycloak-2, physical addresses are [10.244.0.27:7600]
...

13. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

thradec Mar 15, 2019 2:05 AM (in response to tdarimont)

Hi all

@Sam no I'm on onprem k8s cluster
@Thomas that sounds promising, I'll try it and let you know the result, thanks

Regards T.
Actions
14. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

sebastian.laskawiec Mar 15, 2019 5:42 AM (in response to thradec)

Hmmm, that's very interesting! I was almost sure JGroups would eliminate such a node from the view. belaban What do you think about this?

The easiest way to check the current view it to "grep" your logs and look for "view" word. The last one will be your view.
Actions

1 2 Previous Next

Go to original post