Listeners on JMS topic not getting messages if not on the same wildfly 11/12 server

[sreckom]

Hello, all.

I'm migrating application from wildfly 8.1 to the latest version (wildfly 12). Application is running on domain mode configuration and is using JMS topic to refresh its in-memory cache. When refresh is required, one of the servers creates JMS message and puts it into JMS topic. All the servers in server group (where application is deployed) pick up the message. This is working ok up to wildfly 10 without special customization of messaging. In wildfly 11 and 12 this isn't working any more - only the server that produced the messsage gets it from JMS topic. All other servers don't get this message. I took default domain.xml configuration and did this:

/server-group=main-server-group:write-attribute(name=socket-binding-group,value=full-ha-sockets)

/server-group=main-server-group:write-attribute(name=profile,value=full-ha)

I created topic and connection factory that are used by application

/profile=full-ha/subsystem=messaging-activemq/server=default/jms-topic=ActivatorCacheTopic:add(entries=["java:/si.telekom.activator.cache.topic"])

/profile=full-ha/subsystem=messaging-activemq/server=default/connection-factory=RemoteActivatorConnectionFactory:add(connectors=["http-connector"],entries=["java:jboss/exported/si.telekom.jms.activator.QueueConnectionFactory"])

I run two wildfly servers (server-one, server-two) on the same machine (Centos 7 or windows 10). If I go to server-one to generate the message, only the listener on server-one gets it and vice versa. If I do the same on older wildfly versions, both servers get the message.

So my question is if this is a configuration issue or a bug. I've tried the same on JBoss EAP 7.1 with the same result  - other server doesn't get the message.

Thank you in advance for the answers.

Best regards, Srecko.

[mnovak]

Hi,

I'm suspecting that Artemis cluster did not form up. Could you check in CLI the number of (Artemis) cluster nodes? CLI command should look like:

/host=master/server=server-one/subsystem=messaging-activemq/server=default/cluster-connection=my-cluster:get-nodes()

Also do you see any warning/error messages relates to org.apache.activemq.artemis module in server.log in $JBOSS_HOME/domain/servers/server-one/log

Thanks,

Mirek

[sreckom]

Hello.

I've tried this command and the response looks like this.

[domain@localhost:9990 /] /host=master/server=server-one/subsystem=messaging-activemq/server=default/cluster-connection=my-cluster:get-nodes

{

    "outcome" => "success",

    "result" => {"f013bb53-3ef3-11e8-bff6-0a0027000005" => "/127.0.0.1:8230"}

}

[domain@localhost:9990 /] /host=master/server=server-two/subsystem=messaging-activemq/server=default/cluster-connection=my-cluster:get-nodes

{

    "outcome" => "success",

    "result" => {"ee571778-3ef3-11e8-94d4-38d5471096f8" => "/127.0.0.1:8080"}

}

The only error that I saw regarding artemis was the error about security (since this was a fresh install). I disabled the security and restarted the servers and the log is now clean. The replication of a message still doesn't work. My last test was on wildfly 12.

.

Best regards, Srecko.

[sreckom]

I managed to solve this. The problem was at first a bad reproducer that gave false results (connection factory wasn't found because newer version of wildfly requires java:/jboss/exported/ prefix and I didn't see this error). After solving this I discovered that if I have two wildfly servers running on the same machine, the replication works. If I had servers running on separate machines, the replication didn't work. Digging into this revealed that default configuration of jgroups uses private interface. After directing those socket bindings to public it started working. So if anyone will have similar issues, they are solved by

/socket-binding-group=full-ha-sockets/socket-binding=jgroups-udp:write-attribute(name=interface,value=public)

/socket-binding-group=full-ha-sockets/socket-binding=jgroups-tcp:write-attribute(name=interface,value=public)

/socket-binding-group=full-ha-sockets/socket-binding=jgroups-mping:write-attribute(name=interface,value=public)

Best regards, Srecko.

[mnovak]

Good, you figure it out :-)