Wildfly cluster node not able to retrieve JMS messages from other nodes (standalone mode)

[bpogace]

Hello,
I have two machines running instances of Wildfly 10.1 in standalone mode. I have clustered them using mod_cluster and can access the application deployed in both of them (round robin).
Reading a bit, I have seen that in Wildfly it is possible to have JMS messages synchronizing automatically when the servers are in the same cluster.

The problem is that when I access the application on node 1 and check from the application the queued messages, it will show me only the messages submitted from the same node. If I access the application on node 2 it will show me only the messages submitted from the application running on node 2. So it seems the messages are not being synchronized between the servers, and both servers cannot see queued messages other than "theirs".

The following is the messaging configuration (identical for both servers):

<subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
            <server name="default">
                <security enabled="false"/>
                <cluster user="${jboss.messaging.cluster.user:user}" password="${jboss.messaging.cluster.password:pass}"/>
                <journal type="NIO"/>     
                <bindings-directory path="${path-live}/bindings" relative-to="jms.path.live"/>
                <journal-directory path="${path-live}/journal" relative-to="jms.path.live"/>
                <large-messages-directory path="${path-live}/largemessages" relative-to="jms.path.live"/>
                <paging-directory path="${path-live}/paging" relative-to="jms.path.live"/>
                <security-setting name="#">
                    <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
                    <role name="jms" send="true" consume="true"/>
                </security-setting>
                <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>
                <remote-connector name="netty" socket-binding="messaging"/>
                <remote-connector name="netty-throughput" socket-binding="messaging-throughput">
                    <param name="batch-delay" value="50"/>
                </remote-connector>
                <in-vm-connector name="in-vm" server-id="0"/>
                <remote-acceptor name="netty" socket-binding="messaging"/>
                <remote-acceptor name="netty-throughput" socket-binding="messaging-throughput">
                    <param name="batch-delay" value="50"/>
                    <param name="direct-deliver" value="false"/>
                </remote-acceptor>
                <in-vm-acceptor name="in-vm" server-id="0"/>
                <broadcast-group name="bg-group1" jgroups-channel="activemq-cluster" connectors="netty"/>
                <discovery-group name="dg-group1" jgroups-channel="activemq-cluster" refresh-timeout="1000"/>
                <cluster-connection name="my-cluster" address="jms" connector-name="netty" message-load-balancing-type="STRICT" discovery-group="dg-group1"/>
                <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
                <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
                <jms-queue name="RunCommandQueue" entries="java:/jms/RunCommandQueue java:jboss/exported/jms/RunCommandQueue"/>
                <jms-queue name="ServiceCommandQueue" entries="java:/jms/ServiceCommandQueue java:jboss/exported/jms/ServiceCommandQueue"/>
                <jms-queue name="EventQueue" entries="java:/jms/EventQueue java:jboss/exported/jms/EventQueue"/>
                <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
                <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="netty" ha="true" block-on-acknowledge="true" reconnect-attempts="-1"/>
                <connection-factory name="EM2Factory" entries="java:jboss/exported/jms/EM2Factory" connectors="netty" ha="true"/>
                <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory java:/jms/EM2Factory java:/jmsRunCommandQueue java:/jms/ServiceCommandQueue java:/jms/EventQueue" connectors="netty" transaction="xa" reconnect-attempts="-1"/>
            </server>
        </subsystem>

Here the socket bindings:

<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
        <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
        <socket-binding name="ajp" port="${jboss.ajp.port:14126}"/>
        <socket-binding name="http" port="${jboss.http.port:8080}"/>
        <socket-binding name="https" port="${jboss.https.port:8443}"/>
        <socket-binding name="iiop" interface="unsecure" port="3528"/>
        <socket-binding name="iiop-ssl" interface="unsecure" port="3529"/>
        <socket-binding name="jgroups-mping" interface="private" port="0" multicast-address="${jboss.default.multicast.address:224.0.1.105}" multicast-port="45700"/>
        <socket-binding name="jgroups-tcp" interface="private" port="7600"/>
        <socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>
        <socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:224.0.1.105}" multicast-port="45688"/>
        <socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>
        <socket-binding name="modcluster" port="0" multicast-address="${jboss.default.multicast.address:224.0.1.105}" multicast-port="23364"/>
        <socket-binding name="txn-recovery-environment" port="4712"/>
        <socket-binding name="txn-status-manager" port="4713"/>
        <socket-binding name="messaging" port="5445"/>
        <socket-binding name="messaging-throughput" port="5455"/>
        <outbound-socket-binding name="mail-smtp">
            <remote-destination host="localhost" port="25"/>
        </outbound-socket-binding>
    </socket-binding-group>

Any help would be great since I cannot seem to find the correct configuration.
Thank you.

[mnovak]

Hi Besian,

I think that there is misunderstanding how Artemis cluster works. What you see is actually expected behavior as messages are distributed to all nodes in cluster. So if you send 10 messages to cluster with 2 nodes then messages are round robined to all nodes so you should see 5 messages on 1st node and 5 messages on 2nd node.

Artemis cluster does not work like that messages would be synced on all nodes. Each node contains just some part of messages for the given queue. This scales much better.

I can see that you've specified:

<cluster-connection name="my-cluster" ... message-load-balancing-type="STRICT" ... /> 

which strictly load balances messages to all nodes in cluster but does not allow messages to be redistributed to other nodes in cluster where is starving consumer. Set message-load-balancing-type="ON_DEMAND" to allow message redistribution. This option will cause that messages will be load balanced only to nodes with active consumers and in case that some consumer disconnects, it will allow messages to be redistributed to another node with active consumer.

Thanks,

Mirek

[bpogace]

Hi Mirek,

Thank you for the reply. I understand now why I couldn't find a simple configuration for solving my problem.

The application I am trying to deploy needs the configuration I described in order to work. Just to give you an example, there is a single administrator user which should be capable of managing all messages present in the queues of the running application. Since he may login on different nodes, the messages need to be consistent (synchronized) between them.

Is there any way of doing this with Artemis or generally with Wildfly? From your reply I'm not having the feeling there is. In that case I would need to specify a single node for managing JMS resources, and so I could rely on a live-backup architecture for achieving HA. Am I correct?

Best Regards,
Besian.

[mnovak]

I'm not aware of such feature in Artemis or in Wildfly. You will need to connect to CLI of each node in cluster and call /subsystem=messaging-activemq/server=default/jms-queue=InQueue:count-messages on each queue. This might be a script for this purpose.

And yes, if there is just one server Widfly/Artemis then there is not such problem.