8 Replies Latest reply on Feb 28, 2019 4:14 PM by stephan_b

    Isolating multicast cross-talk w/ WildFly clusters

    stephan_b

      I have two WildFly clusters (8.2.1, sorry, cannot be helped right now) on one subnet. Each cluster uses a different -u multicast address, yet there is still cross-talk.

      What else need to be made unique so the cross-talk will stop?

        • 1. Re: Isolating multicast cross-talk w/ WildFly clusters
          rhn-support-abhati

          Hi,

           

          What is the command you use to start JBoss clusters ?

          Can you verify in server logs if both clusters started on different multicast address or not.

           

          You can switch to TCP based cluster as well to avoid cross talk among clusters.

          • 2. Re: Isolating multicast cross-talk w/ WildFly clusters
            stephan_b

            Here is the command from the server.log file:

             

            sun.java.command = E:\Programs\wildfly-8.2.1.Final\jboss-modules.jar -mp E:\Programs\wildfly-8.2.1.Final\modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=E:\Programs\wildfly-8.2.1.Final -c standalone-full-ha.xml -b 10.xxx.xxx.55 -u 230.0.11.4 -Djboss.node.name=iamnode1 -Djgroups.bind_addr=10.xxx.xxx.55 -Dnete.j2ee.vendor=jboss -Djava.net.preferIPv4Stack=true -Djboss.as.management.blocking.timeout=900

             

            Yes, both clusters start on a different multicast address.

             

            I can't do unicast. I don't know if this is a WildFly issue or application issue, but my software vendor stated that unless I wanted to do shared storage for clustering (which I don't), I should run multicast replication.

            • 3. Re: Isolating multicast cross-talk w/ WildFly clusters
              pferraro

              Can you elaborate on what you mean by crosstalk?  Do your cluster memberships look correct?  What evidence do you have that messages sent by members of one cluster are being received by members of the other cluster?  Can you paste relevant log messages?

              • 4. Re: Isolating multicast cross-talk w/ WildFly clusters
                stephan_b

                The log messages appear to come through the application that is deployed on WildFly, but are from HornetQ:

                 

                2019-02-27 16:53:03,050 ERROR [ims.tmt.persistence] (Thread-4 (HornetQ-client-global-threads-137232667)) PersistenceProvider: JMS:ID:0e5c44ba-3ada-11e9-a4f2-59349a6b8c8d: PersistenceProvider:: lockObject: ps execute returns 0

                2019-02-27 16:53:03,097 WARN  [ims.tmt.EventMessageWrapper] (Thread-4 (HornetQ-client-global-threads-137232667)) initNewStyle: Couldn't find or lock event 1abebcb2-2affb3b0-294d9b83-c9ee07

                2019-02-27 16:53:03,097 ERROR [ims.tmt.IMSMessageListener] (Thread-4 (HornetQ-client-global-threads-137232667)) onMessage: No event found in message ID:0e5c44ba-3ada-11e9-a4f2-59349a6b8c8d

                 

                This is really bizarre at this point, because I just reconfigured both cluster sets to use

                <subsystem xmlns="urn:jboss:domain:jgroups:2.0" default-stack="tcp">

                Before that, the default-stack was udp. I can see DEBUG level jgroups messages indicating the clusters now communicate over TCP, but there is STILL cross-talk that leads to the above-mentioned log messages. Is there anything else I need to do other than change default-stack to from udp to tcp?

                • 5. Re: Isolating multicast cross-talk w/ WildFly clusters
                  stephan_b

                  It gets even more weird. When I shut down a node from cluster 2, I get the following logged in all 3 members of cluster 1:

                   

                  2019-02-27 20:27:10,290 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from /10.xxx.xxx.110:63364. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

                  • 6. Re: Isolating multicast cross-talk w/ WildFly clusters
                    stephan_b

                    I was able to install a clean unicast cluster node so I could compare standalone-full-ha.xml files, and one of the culprits seems to be that my tcp stack was using MPING as a method, vs TCPPING, and I have to change over some broadcast-group and discovery-groups configurations.

                     

                    Looks like I have some tweaking to do on my existing clusters.

                    • 7. Re: Isolating multicast cross-talk w/ WildFly clusters
                      pferraro

                      The -u option corresponds to the jboss.default.multicast.address system property.  In WF 8.2, this is consumed by both the MPING protocol from the tcp stack, and the multicast address of the UDP transport from the udp stack.  If you have multiple channels trying to use both stacks at once, you would have conflicts.

                      1 of 1 people found this helpful
                      • 8. Re: Isolating multicast cross-talk w/ WildFly clusters
                        stephan_b

                        Thanks everyone. I was able to switch my cluster members to unicast and tcpping. So far so good.