-
1. Re: Isolating multicast cross-talk w/ WildFly clusters
rhn-support-abhati Feb 27, 2019 3:41 AM (in response to stephan_b)Hi,
What is the command you use to start JBoss clusters ?
Can you verify in server logs if both clusters started on different multicast address or not.
You can switch to TCP based cluster as well to avoid cross talk among clusters.
-
2. Re: Isolating multicast cross-talk w/ WildFly clusters
stephan_b Feb 27, 2019 11:01 AM (in response to rhn-support-abhati)Here is the command from the server.log file:
sun.java.command = E:\Programs\wildfly-8.2.1.Final\jboss-modules.jar -mp E:\Programs\wildfly-8.2.1.Final\modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=E:\Programs\wildfly-8.2.1.Final -c standalone-full-ha.xml -b 10.xxx.xxx.55 -u 230.0.11.4 -Djboss.node.name=iamnode1 -Djgroups.bind_addr=10.xxx.xxx.55 -Dnete.j2ee.vendor=jboss -Djava.net.preferIPv4Stack=true -Djboss.as.management.blocking.timeout=900
Yes, both clusters start on a different multicast address.
I can't do unicast. I don't know if this is a WildFly issue or application issue, but my software vendor stated that unless I wanted to do shared storage for clustering (which I don't), I should run multicast replication.
-
3. Re: Isolating multicast cross-talk w/ WildFly clusters
pferraro Feb 27, 2019 12:15 PM (in response to stephan_b)Can you elaborate on what you mean by crosstalk? Do your cluster memberships look correct? What evidence do you have that messages sent by members of one cluster are being received by members of the other cluster? Can you paste relevant log messages?
-
4. Re: Isolating multicast cross-talk w/ WildFly clusters
stephan_b Feb 27, 2019 4:58 PM (in response to pferraro)The log messages appear to come through the application that is deployed on WildFly, but are from HornetQ:
2019-02-27 16:53:03,050 ERROR [ims.tmt.persistence] (Thread-4 (HornetQ-client-global-threads-137232667)) PersistenceProvider: JMS:ID:0e5c44ba-3ada-11e9-a4f2-59349a6b8c8d: PersistenceProvider:: lockObject: ps execute returns 0
2019-02-27 16:53:03,097 WARN [ims.tmt.EventMessageWrapper] (Thread-4 (HornetQ-client-global-threads-137232667)) initNewStyle: Couldn't find or lock event 1abebcb2-2affb3b0-294d9b83-c9ee07
2019-02-27 16:53:03,097 ERROR [ims.tmt.IMSMessageListener] (Thread-4 (HornetQ-client-global-threads-137232667)) onMessage: No event found in message ID:0e5c44ba-3ada-11e9-a4f2-59349a6b8c8d
This is really bizarre at this point, because I just reconfigured both cluster sets to use
<subsystem xmlns="urn:jboss:domain:jgroups:2.0" default-stack="tcp">
Before that, the default-stack was udp. I can see DEBUG level jgroups messages indicating the clusters now communicate over TCP, but there is STILL cross-talk that leads to the above-mentioned log messages. Is there anything else I need to do other than change default-stack to from udp to tcp?
-
5. Re: Isolating multicast cross-talk w/ WildFly clusters
stephan_b Feb 27, 2019 8:30 PM (in response to stephan_b)It gets even more weird. When I shut down a node from cluster 2, I get the following logged in all 3 members of cluster 1:
2019-02-27 20:27:10,290 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from /10.xxx.xxx.110:63364. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
-
6. Re: Isolating multicast cross-talk w/ WildFly clusters
stephan_b Feb 27, 2019 9:01 PM (in response to stephan_b)I was able to install a clean unicast cluster node so I could compare standalone-full-ha.xml files, and one of the culprits seems to be that my tcp stack was using MPING as a method, vs TCPPING, and I have to change over some broadcast-group and discovery-groups configurations.
Looks like I have some tweaking to do on my existing clusters.
-
7. Re: Isolating multicast cross-talk w/ WildFly clusters
pferraro Feb 28, 2019 9:07 AM (in response to stephan_b)1 of 1 people found this helpfulThe -u option corresponds to the jboss.default.multicast.address system property. In WF 8.2, this is consumed by both the MPING protocol from the tcp stack, and the multicast address of the UDP transport from the udp stack. If you have multiple channels trying to use both stacks at once, you would have conflicts.
-
8. Re: Isolating multicast cross-talk w/ WildFly clusters
stephan_b Feb 28, 2019 4:14 PM (in response to stephan_b)Thanks everyone. I was able to switch my cluster members to unicast and tcpping. So far so good.