1 2 Previous Next 19 Replies Latest reply on Apr 9, 2019 5:55 AM by cablespaghetti

    WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

    thradec

      Hi,

      I'm trying to run keycloak 4.8.3 in kubernetes cluser and I noticed following warning in logs

       

      WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message

       

      basically, I'm using a slightly modified helm chart charts/stable/keycloak at master · helm/charts · GitHub  which use DNS_PING discovery protocol, here is the snippet from standalone-ha.xml

       

              <subsystem xmlns="urn:jboss:domain:jgroups:6.0">
                  <channels default="ee">
                      <channel name="ee" stack="tcp" cluster="ejb"/>
                  </channels>
                  <stacks>
                      <stack name="tcp">
                          <transport type="TCP" socket-binding="jgroups-tcp"/>
                          <protocol type="dns.DNS_PING">
                              <property name="dns_query">
                                  keycloak-headless.ph00a1-sso.svc.cluster.local
                              </property>
                          </protocol>
                          <protocol type="MERGE3"/>
                          <protocol type="FD_SOCK"/>
                          <protocol type="FD_ALL"/>
                          <protocol type="VERIFY_SUSPECT"/>
                          <protocol type="pbcast.NAKACK2"/>
                          <protocol type="UNICAST3"/>
                          <protocol type="pbcast.STABLE"/>
                          <protocol type="pbcast.GMS"/>
                          <protocol type="MFC"/>
                          <protocol type="FRAG3"/>
                      </stack>
      

       

      The warning in the logs appears when some pod is deleted and new one is started, eg. during rolling new version. Before that, everything looks fine and the cluster is formed propely (at least it seems from logs, eg.)

       

       [0m [0m13:31:01,512 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000078: Starting JGroups channel ejb
       [0m [0m13:31:01,512 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000078: Starting JGroups channel ejb
       [0m [0m13:31:01,519 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|1] (2) [keycloak-0, keycloak-1]
       [0m [0m13:31:01,519 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-2) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|1] (2) [keycloak-0, keycloak-1]
       [0m [0m13:31:01,612 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000079: Channel ejb local address is keycloak-1, physical addresses are [10.236.68.237:7600]
       [0m [0m13:31:01,612 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Channel ejb local address is keycloak-1, physical addresses are [10.236.68.237:7600]
       [0m [0m13:31:01,720 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-2) ISPN000128: Infinispan version: Infinispan 'Estrella Galicia' 9.3.1.Final
      
      

       

      The only similar issue which I found is [ISPN-7489] org.jgroups.protocols.TCP emits errors when node leaves the cluster - JBoss Issue Tracker , but it should be fixed already.

       

      I'd be glad for any advice.

      Thanks a lot

      Tomas

        • 1. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
          belaban

          When you're execing into one of the pods, what's the output of 'dig keycloak-headless.ph00a1-sso.svc.cluster.local'?

          Are you running this on minikube/minishift? In my own testing, I found that minikube's DNS service sometimes delays DNS responses. I'm not too familiar with minikube, but it seemed to be running 2 DNS services, instead of 1.

          Please also want to experiment with setting dns_address, see [1] for reference.

          [1] Reliable group communication with JGroups

          • 2. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
            thradec

            Hi and thanks for the reply

             

            this is the output from dig cmd

             

            [jboss@keycloak-0 ~]$ dig keycloak-headless.ph00a1-sso.svc.cluster.local
            
            
            ; <<>> DiG 9.9.4-RedHat-9.9.4-73.el7_6 <<>> keycloak-headless.ph00a1-sso.svc.cluster.local
            ;; global options: +cmd
            ;; Got answer:
            ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12215
            ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
            
            
            ;; OPT PSEUDOSECTION:
            ; EDNS: version: 0, flags:; udp: 4096
            ;; QUESTION SECTION:
            ;keycloak-headless.ph00a1-sso.svc.cluster.local. IN A
            
            
            ;; ANSWER SECTION:
            keycloak-headless.ph00a1-sso.svc.cluster.local. 1 IN A 10.236.68.37
            keycloak-headless.ph00a1-sso.svc.cluster.local. 1 IN A 10.236.66.133
            keycloak-headless.ph00a1-sso.svc.cluster.local. 1 IN A 10.236.69.182
            
            
            ;; Query time: 0 msec
            ;; SERVER: 10.236.0.10#53(10.236.0.10)
            ;; WHEN: Fri Mar 08 12:38:22 UTC 2019
            ;; MSG SIZE  rcvd: 123

             

            the application is running in cluster build by RKE Rancher Kubernetes Engine, the following is the list and version of used components

             

              kubernetes:  rancher-hyperkube:v1.12.3-rancher1
              etcd:  rancher-coreos-etcd:v3.2.24
              alpine:  rancher-rke-tools:v0.1.16
              nginx_proxy:  rancher-rke-tools:v0.1.16
              cert_downloader:  rancher-rke-tools:v0.1.16
              kubernetes_services_sidecar:  rancher-rke-tools:v0.1.16
              kubedns:  rancher-k8s-dns-kube-dns-amd64:1.14.13
              dnsmasq:  rancher-k8s-dns-dnsmasq-nanny-amd64:1.14.13
              kubedns_sidecar:  rancher-k8s-dns-sidecar-amd64:1.14.13
              kubedns_autoscaler:  rancher-cluster-proportional-autoscaler-amd64:1.0.0
              pod_infra_container:  rancher-pause-amd64:3.1
            • 3. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
              sebastian.laskawiec

              So it seems Kubernetes killed a Pod and JGroups didn't noticed that it already disappeared.

               

              If you're running Kubernetes Rolling Update with more than 2 nodes (I see you're running 3), you should be good and this should do no harm to you.

               

              There are two things that can cause this that come into my mind:

              1. FD_* protocols didn't notice the change. The killed pod should be already removed from the view.
              2. logical_addr_cache_expiration is still too high in the TP.

               

              belaban What do you think about this?

               

              It would be great if you could help us diagnosing this Tomas. Unfortunately, both require changes in the xml configuration.

               

              The latter is much easier to verify and I highly recommend to start with that. Here's a configuration snippet:

              <transport type="TCP" socket-binding="jgroups-tcp" >

                  <!-- This needs to be something lower than 360000, maybe 260000? -->

                  <property name="logical_addr_cache_expiration">260000</property>

              </transport>

               

              As for the former, you may take a look at one of my old configuration tuning guides: presentations/README.asciidoc at master · slaskawi/presentations · GitHub

              In essence, you'd need to remove VERIFY_SUSPECT and tune FD_ALL settings. You may start with copy-pasting things I have there.

              • 4. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
                thradec

                Hi, thanks for the hints, I'd be glad to help you find the issue, I'll try to change the xml configuration and let you know if it helps

                • 5. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
                  thradec

                  Hi, I tried to set logical_addr_cache_expiration property, but the warning is still there, I'll try the others settings, which you mentioned.

                  • 6. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
                    cablespaghetti

                    We have this exact same issue using that same Helm chart (v4.4.0) with Keycloak 4.8.3. It does seem to have appeared after we restarted all the containers. Any progress with working out the fix?

                     

                    edit: I've now added this to my Helm values file, to disable Java's idiotic infinite DNS caching. This seems to have fixed the problem for me although only time will tell...

                      ## Additional arguments to start command e.g. -Dkeycloak.import= to load a realm
                      extraArgs: "-Dsun.net.inetaddr.ttl=0"
                    
                    • 7. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
                      sebastian.laskawiec

                      Unfortunately no progress Sam, Tomas. I tried to reproduce this locally a number of times and it seems this has something to do with your environment.

                       

                      Sam's idea about lowering TTL on DNS cache might work. As I said before, the scenario is that a node disappears (Kubernetes kills it) and JGroups don't notice that on time.

                       

                      I also repeat what I said before - unless you have a really weird setup, this will do no harm. JGroups should update the view shortly after this warning is emitted.

                      • 8. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
                        cablespaghetti

                        Unfortunately my "fix" stopped working again after I left work yesterday. The error keeps printing every minute or so forever, so it's not a temporary thing until JGroups updates.

                        • 9. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
                          thradec

                          Hi, neither I have any positive progress.

                           

                          - I tried some other configuration changes (eg. remove VERIFY_SUSPECT and set timeouts in FD_ALL), but the cluster start to be even more unstable, so I had to revert

                          - I can confirm the same behavior as Sam, that the warning is permanent and doesn't disappear after a while

                          - but luckily I didn't observer any negative influence on functionality

                           

                          Can I somehow check the current view of JGroups?

                          • 10. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
                            cablespaghetti

                            Tomas. Are you running on AWS by any chance? We're using EKS with the AWS VPC CNI Plugin. I notice that JGroups uses multicast for some of its functionality...I wonder if Amazon's lack of multicast support is causing this...

                            • 11. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
                              tdarimont

                              Hello,

                               

                              I see this problem too in a 3 node Keycloak cluster, which I created with the Keycloak Helm Chart running on Azure AKS. As test I deleted the keycloak-2 pod, which is then automatically recreated.

                              The cluster seems to behave normally, but one can see the mentioned warn messages in the log:

                              
                              keycloak-1 keycloak 17:22:48,312 INFO  [org.infinispan.CLUSTER] (remote-thread--p17-t1) [Context=client-mappings] ISPN100002: Starting rebalance with members [keycloak-1, keycloak-0, keycloak-2], phase READ_OLD_WRITE_ALL, topology id 27
                              keycloak-1 keycloak 17:22:49,814 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=work] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 28
                              keycloak-1 keycloak 17:22:50,090 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=work] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 29
                              keycloak-1 keycloak 17:22:50,101 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=sessions] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 47
                              keycloak-1 keycloak 17:22:50,103 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=offlineSessions] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 47
                              keycloak-1 keycloak 17:22:50,104 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=actionTokens] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 48
                              keycloak-1 keycloak 17:22:50,106 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=clientSessions] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 46
                              keycloak-0 keycloak 18:46:10,946 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:46:45,085 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:47:19,978 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:47:48,271 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:48:17,167 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:49:00,858 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:49:40,540 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:22:50,391 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t23) [Context=clientSessions] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 47
                              keycloak-1 keycloak 17:22:50,391 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t22) [Context=actionTokens] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 49
                              keycloak-1 keycloak 17:22:50,391 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t30) [Context=sessions] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 48
                              keycloak-1 keycloak 17:22:50,392 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=work] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 30
                              keycloak-1 keycloak 17:22:50,392 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t28) [Context=offlineSessions] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 48
                              keycloak-0 keycloak 18:50:18,760 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:50:55,692 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:51:34,666 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:22:50,394 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=offlineClientSessions] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 49
                              keycloak-1 keycloak 17:22:50,395 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=authenticationSessions] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 48
                              keycloak-0 keycloak 18:52:00,242 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:52:18,624 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:52:40,562 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak-1 keycloak 18:52:58,724 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak 17:22:50,399 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t28) [Context=loginFailures] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 48
                              keycloak-1 keycloak 17:22:50,692 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t27) [Context=sessions] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 49
                              keycloak-1 keycloak 17:22:50,693 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t22) [Context=offlineClientSessions] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 50
                              keycloak-1 keycloak 17:22:50,693 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t25) [Context=authenticationSessions] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 49
                              keycloak-1 keycloak 17:22:50,693 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t23) [Context=clientSessions] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 48
                              keycloak-1 keycloak 17:22:50,692 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t30) [Context=offlineSessions] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 49
                              keycloak-1 keycloak 17:22:50,692 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t26) [Context=loginFailures] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 49
                              keycloak-1 keycloak 17:22:50,697 INFO  [org.infinispan.CLUSTER] (remote-thread--p17-t1) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 28
                              keycloak-1 keycloak 17:22:50,696 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t28) [Context=actionTokens] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 50
                              keycloak-1 keycloak 17:22:50,993 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t26) [Context=offlineClientSessions] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 51
                              keycloak-1 keycloak 17:22:50,993 INFO  [org.infinispan.CLUSTER] (remote-thread--p17-t1) [Context=client-mappings] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 29
                              keycloak-1 keycloak 17:22:50,993 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t28) [Context=authenticationSessions] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 50
                              keycloak-1 keycloak 17:22:50,993 INFO  [org.infinispan.CLUSTER] (remote-thread--p5-t30) [Context=loginFailures] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 50
                              keycloak-0 keycloak 18:53:23,710 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:53:40,948 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:54:17,090 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:55:01,945 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:55:28,188 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:56:01,495 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:56:43,653 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:56:59,333 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:57:24,373 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:57:49,312 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:58:09,373 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:58:52,386 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 18:59:35,883 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:00:13,669 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:00:55,143 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:01:17,637 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:01:53,316 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:02:21,274 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:02:45,114 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:03:19,536 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:03:35,989 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:03:51,034 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:22:51,294 INFO  [org.infinispan.CLUSTER] (remote-thread--p17-t1) [Context=client-mappings] ISPN100010: Finished rebalance with members [keycloak-1, keycloak-0, keycloak-2], topology id 30
                              keycloak-1 keycloak 17:28:58,859 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloakkeycloak-0  17:29:35,708 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:30:00,904 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:30:45,616 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:31:10,447 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:31:48,612 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:32:19,204 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak keycloak 17:32:43,272 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:33:06,470 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:33:23,396 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 19:04:09,147 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:04:50,647 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:05:05,835 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:05:29,407 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:06:12,327 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak-0 keycloak 19:06:54,637 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-0) JGRP000032: keycloak-0: no physical address for keycloak-2, dropping message
                              keycloak 17:34:01,096 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:34:26,468 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:34:45,733 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:35:11,125 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:35:51,240 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:36:07,178 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:36:28,214 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:37:06,967 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:37:22,194 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:37:57,296 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:38:41,005 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:39:11,305 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:39:30,659 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:40:12,674 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:40:51,718 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:41:06,955 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:41:43,394 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:42:26,488 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:43:00,361 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:43:25,884 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:43:53,646 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:44:08,871 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:44:41,504 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:45:09,314 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:45:53,227 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:46:21,455 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:47:00,441 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:47:32,014 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:48:10,429 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:48:34,421 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:49:08,532 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:49:50,324 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:50:20,557 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:50:51,530 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:51:34,304 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message
                              keycloak-1 keycloak 17:52:09,189 WARN  [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-1) JGRP000032: keycloak-1: no physical address for keycloak-2, dropping message

                              The JGroups config looks like this:

                                      <subsystem xmlns="urn:jboss:domain:jgroups:6.0">
                                          <channels default="ee">
                                              <channel name="ee" stack="tcp" cluster="ejb"/>
                                          </channels>
                                          <stacks>
                                              <stack name="udp">
                                                  <transport type="UDP" socket-binding="jgroups-udp"/>
                                                  <protocol type="dns.DNS_PING">
                                                      <property name="dns_query">
                                                          keycloak-headless.default.svc.cluster.local
                                                      </property>
                                                  </protocol>
                                                  <protocol type="MERGE3"/>
                                                  <protocol type="FD_SOCK"/>
                                                  <protocol type="FD_ALL"/>
                                                  <protocol type="VERIFY_SUSPECT"/>
                                                  <protocol type="pbcast.NAKACK2"/>
                                                  <protocol type="UNICAST3"/>
                                                  <protocol type="pbcast.STABLE"/>
                                                  <protocol type="pbcast.GMS"/>
                                                  <protocol type="UFC"/>
                                                  <protocol type="MFC"/>
                                                  <protocol type="FRAG3"/>
                                              </stack>
                                              <stack name="tcp">
                                                  <transport type="TCP" socket-binding="jgroups-tcp"/>
                                                  <protocol type="dns.DNS_PING">
                                                      <property name="dns_query">
                                                          keycloak-headless.default.svc.cluster.local
                                                      </property>
                                                  </protocol>
                                                  <protocol type="MERGE3"/>
                                                  <protocol type="FD_SOCK"/>
                                                  <protocol type="FD_ALL"/>
                                                  <protocol type="VERIFY_SUSPECT"/>
                                                  <protocol type="pbcast.NAKACK2"/>
                                                  <protocol type="UNICAST3"/>
                                                  <protocol type="pbcast.STABLE"/>
                                                  <protocol type="pbcast.GMS"/>
                                                  <protocol type="MFC"/>
                                                  <protocol type="FRAG3"/>
                                              </stack>
                                          </stacks>
                                      </subsystem>
                              
                                      

                               

                              Cheers,

                              Thomas

                              • 12. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
                                tdarimont

                                Quick update: I just tried dns.DNS_PING with Keycloak 4.8.3.Final and the latest jgroups 4.0.19 (which was just released today ;-) ) on my 3-node Kubernetes Cluster running on Azure AKS.

                                (I added the new jgroups-4.0.19.Final.jar to the jgroups/main module and updated the module.xml to refer to the new jar).

                                 

                                With jgroups-4.0.19.Final.jar I don't see error messages like "no physical address for keycloak-2, dropping message" anymore

                                 

                                So it seems that upgrading the jgroups version helps to get rid of this problem - note, that I did not change the dns.DNS_PING configuration.

                                 

                                I saw that the issue [JGRP-2316] DNS_PING is not using correct ports with SRV based service discovery - JBoss Issue Tracker  is already fixed with 4.0.16, so I guess one could also try that version.

                                 

                                 

                                # Killed Pod keycloak-2
                                
                                
                                keycloak-2 keycloak *** JBossAS process (228) received TERM signal ***
                                keycloak-0 keycloak 22:27:58,030 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t15) [Context=sessions] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 23
                                keycloak-1 keycloak 22:27:58,031 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|4] (2) [keycloak-0, keycloak-1]
                                keycloak-0 keycloak 22:27:58,034 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t11) [Context=loginFailures] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 22
                                keycloak-0 keycloak 22:27:58,038 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t13) [Context=offlineSessions] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 23
                                keycloak-1 keycloak 22:27:58,038 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN100001: Node keycloak-2 left the cluster
                                keycloak-1 keycloak 22:27:58,041 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|4] (2) [keycloak-0, keycloak-1]
                                keycloak-0 keycloak 22:27:58,039 INFO  [org.infinispan.CLUSTER] (remote-thread--p17-t3) [Context=client-mappings] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 23
                                keycloak-1 keycloak 22:27:58,043 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN100001: Node keycloak-2 left the cluster
                                keycloak-1 keycloak 22:27:58,044 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|4] (2) [keycloak-0, keycloak-1]
                                keycloak-1 keycloak 22:27:58,045 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN100001: Node keycloak-2 left the cluster
                                keycloak-1 keycloak 22:27:58,045 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|4] (2) [keycloak-0, keycloak-1]
                                keycloak-1 keycloak 22:27:58,045 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN100001: Node keycloak-2 left the cluster
                                keycloak-1 keycloak 22:27:58,046 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|4] (2) [keycloak-0, keycloak-1]
                                keycloak-1 keycloak 22:27:58,046 INFO  [org.infinispan.CLUSTER] (thread-15,ejb,keycloak-1) ISPN100001: Node keycloak-2 left the cluster
                                keycloak-0 keycloak 22:27:58,051 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t10) [Context=authenticationSessions] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 23
                                keycloak-0 keycloak 22:27:58,051 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t15) [Context=clientSessions] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 21
                                keycloak-0 keycloak 22:27:58,052 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t11) [Context=work] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 21
                                keycloak-0 keycloak 22:27:58,052 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t13) [Context=offlineClientSessions] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 24
                                keycloak-0 keycloak 22:27:58,055 INFO  [org.infinispan.CLUSTER] (remote-thread--p3-t13) [Context=actionTokens] ISPN100010: Finished rebalance with members [keycloak-0, keycloak-1], topology id 21
                                - keycloak-2
                                + keycloak-2 › keycloak
                                
                                
                                # Pod keycloak-2 restarts
                                ...
                                keycloak-2 keycloak 22:28:28,219 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 39) WFLYCLJG0001: Activating JGroups subsystem. JGroups version 4.0.19
                                ...
                                keycloak-0 keycloak 22:28:31,121 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
                                keycloak-0 keycloak 22:28:31,121 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN100000: Node keycloak-2 joined the cluster
                                keycloak-0 keycloak 22:28:31,122 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
                                keycloak-0 keycloak 22:28:31,122 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN100000: Node keycloak-2 joined the cluster
                                keycloak-0 keycloak 22:28:31,122 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
                                keycloak-0 keycloak 22:28:31,122 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN100000: Node keycloak-2 joined the cluster
                                keycloak-0 keycloak 22:28:31,130 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
                                keycloak-0 keycloak 22:28:31,134 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN100000: Node keycloak-2 joined the cluster
                                keycloak-0 keycloak 22:28:31,138 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
                                keycloak-0 keycloak 22:28:31,138 INFO  [org.infinispan.CLUSTER] (thread-20,ejb,keycloak-0) ISPN100000: Node keycloak-2 joined the cluster
                                keycloak-1 keycloak 22:28:31,147 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
                                keycloak-1 keycloak 22:28:31,148 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN100000: Node keycloak-2 joined the cluster
                                keycloak-1 keycloak 22:28:31,151 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
                                keycloak-1 keycloak 22:28:31,152 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN100000: Node keycloak-2 joined the cluster
                                keycloak-1 keycloak 22:28:31,152 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
                                keycloak-1 keycloak 22:28:31,153 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN100000: Node keycloak-2 joined the cluster
                                keycloak-1 keycloak 22:28:31,155 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
                                keycloak-1 keycloak 22:28:31,155 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN100000: Node keycloak-2 joined the cluster
                                keycloak-1 keycloak 22:28:31,156 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
                                keycloak-1 keycloak 22:28:31,156 INFO  [org.infinispan.CLUSTER] (thread-16,ejb,keycloak-1) ISPN100000: Node keycloak-2 joined the cluster
                                keycloak-2 keycloak 22:28:31,664 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000078: Starting JGroups channel ejb
                                keycloak-2 keycloak 22:28:31,667 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000078: Starting JGroups channel ejb
                                keycloak-2 keycloak 22:28:31,681 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-2) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
                                keycloak-2 keycloak 22:28:31,681 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-1) ISPN000094: Received new cluster view for channel ejb: [keycloak-0|5] (3) [keycloak-0, keycloak-1, keycloak-2]
                                keycloak-2 keycloak 22:28:31,701 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000079: Channel ejb local address is keycloak-2, physical addresses are [10.244.0.27:7600]
                                keycloak-2 keycloak 22:28:31,713 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Channel ejb local address is keycloak-2, physical addresses are [10.244.0.27:7600]
                                ...
                                • 13. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
                                  thradec

                                  Hi all

                                   

                                  @Sam no I'm on onprem k8s cluster

                                  @Thomas that sounds promising, I'll try it and let you know the result, thanks

                                   

                                  Regards T.

                                  • 14. Re: WARN [org.jgroups.protocols.TCP] (TQ-Bundler-7,ejb,keycloak-2) JGRP000032: keycloak-2: no physical address for keycloak-1, dropping message
                                    sebastian.laskawiec

                                    Hmmm, that's very interesting! I was almost sure JGroups would eliminate such a node from the view. belaban What do you think about this?

                                     

                                    The easiest way to check the current view it to "grep" your logs and look for "view" word. The last one will be your view.

                                    1 2 Previous Next