What is the command you use to start JBoss clusters ?
Can you verify in server logs if both clusters started on different multicast address or not.
You can switch to TCP based cluster as well to avoid cross talk among clusters.
Here is the command from the server.log file:
sun.java.command = E:\Programs\wildfly-8.2.1.Final\jboss-modules.jar -mp E:\Programs\wildfly-8.2.1.Final\modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=E:\Programs\wildfly-8.2.1.Final -c standalone-full-ha.xml -b 10.xxx.xxx.55 -u 18.104.22.168 -Djboss.node.name=iamnode1 -Djgroups.bind_addr=10.xxx.xxx.55 -Dnete.j2ee.vendor=jboss -Djava.net.preferIPv4Stack=true -Djboss.as.management.blocking.timeout=900
Yes, both clusters start on a different multicast address.
I can't do unicast. I don't know if this is a WildFly issue or application issue, but my software vendor stated that unless I wanted to do shared storage for clustering (which I don't), I should run multicast replication.
Can you elaborate on what you mean by crosstalk? Do your cluster memberships look correct? What evidence do you have that messages sent by members of one cluster are being received by members of the other cluster? Can you paste relevant log messages?
The log messages appear to come through the application that is deployed on WildFly, but are from HornetQ:
2019-02-27 16:53:03,050 ERROR [ims.tmt.persistence] (Thread-4 (HornetQ-client-global-threads-137232667)) PersistenceProvider: JMS:ID:0e5c44ba-3ada-11e9-a4f2-59349a6b8c8d: PersistenceProvider:: lockObject: ps execute returns 0
2019-02-27 16:53:03,097 WARN [ims.tmt.EventMessageWrapper] (Thread-4 (HornetQ-client-global-threads-137232667)) initNewStyle: Couldn't find or lock event 1abebcb2-2affb3b0-294d9b83-c9ee07
2019-02-27 16:53:03,097 ERROR [ims.tmt.IMSMessageListener] (Thread-4 (HornetQ-client-global-threads-137232667)) onMessage: No event found in message ID:0e5c44ba-3ada-11e9-a4f2-59349a6b8c8d
This is really bizarre at this point, because I just reconfigured both cluster sets to use
<subsystem xmlns="urn:jboss:domain:jgroups:2.0" default-stack="tcp">
Before that, the default-stack was udp. I can see DEBUG level jgroups messages indicating the clusters now communicate over TCP, but there is STILL cross-talk that leads to the above-mentioned log messages. Is there anything else I need to do other than change default-stack to from udp to tcp?
It gets even more weird. When I shut down a node from cluster 2, I get the following logged in all 3 members of cluster 1:
2019-02-27 20:27:10,290 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from /10.xxx.xxx.110:63364. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
I was able to install a clean unicast cluster node so I could compare standalone-full-ha.xml files, and one of the culprits seems to be that my tcp stack was using MPING as a method, vs TCPPING, and I have to change over some broadcast-group and discovery-groups configurations.
Looks like I have some tweaking to do on my existing clusters.
1 of 1 people found this helpful
The -u option corresponds to the jboss.default.multicast.address system property. In WF 8.2, this is consumed by both the MPING protocol from the tcp stack, and the multicast address of the UDP transport from the udp stack. If you have multiple channels trying to use both stacks at once, you would have conflicts.
Thanks everyone. I was able to switch my cluster members to unicast and tcpping. So far so good.