(ActiveMQ) ClusterConnectionBridge not created in Wildfly 15 cluster (domain)
Hello, I have a following issue with Wildfly 15 cluster (also tried with latest Wildfly 17).
ClusterConnectionBridge is not being created after Wildfly is started, so messages are not being synchronized between servers in the cluster when they are submitted via topic.
This is how it works with Wildfly 14 (log grep for "org.apache.activemq.artemis.core.server"):
2019-08-05 15:32:56,057 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221012: Using AIO Journal 2019-08-05 15:32:56,697 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 954,466,304 2019-08-05 15:32:57,019 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE 2019-08-05 15:32:57,019 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP 2019-08-05 15:32:57,019 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ 2019-08-05 15:32:57,020 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP 2019-08-05 15:32:57,110 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221034: Waiting indefinitely to obtain live lock 2019-08-05 15:32:57,110 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221035: Live Server Obtained live lock 2019-08-05 15:32:58,349 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221007: Server is now live 2019-08-05 15:32:58,350 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.6.3.jbossorg-001 [default, nodeID=ceb7187b-fd8a-11e8-8274-0050568b6fed] 2019-08-05 15:32:59,410 WARN [org.apache.activemq.artemis.core.server] (Thread-2 (ActiveMQ-client-global-threads)) AMQ224078: The size of duplicate cache detection (<id_cache-size>) appears to be too large 20,000. It should be no greater than the number of messages that can be squeezed into conformation buffer () 1,048,576. 2019-08-05 15:32:59,779 INFO [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1426f525)) AMQ221027: Bridge ClusterConnectionBridge@484e51c8 [name=$.artemis.internal.sf.my-cluster.f381572d-fde8-11e8-8ee3-0050568b484f, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.f381572d-fde8-11e8-8ee3-0050568b484f, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=ceb7187b-fd8a-11e8-8274-0050568b6fed], temp=false]@26eea0e targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@484e51c8 [name=$.artemis.internal.sf.my-cluster.f381572d-fde8-11e8-8ee3-0050568b484f, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.f381572d-fde8-11e8-8ee3-0050568b484f, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=ceb7187b-fd8a-11e8-8274-0050568b6fed], temp=false]@26eea0e targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=test-2], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@75434150[nodeUUID=ceb7187b-fd8a-11e8-8274-0050568b6fed, connector=TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=test-1, address=jms, server=ActiveMQServerImpl::serverUUID=ceb7187b-fd8a-11e8-8274-0050568b6fed])) [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=test-2], discoveryGroupConfiguration=null]] is connected
In Wildfly 15 (using same configuration and same application builds), cluster is well formed, ActiveMQ broker is started, but log message with ClusterConnectionBridge is never logged. Any message submitted to a specific topic is NOT broadcasted, only is "received" and processed on a server where it is submitted.
... 08:35:25,535 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/opt/wildfly-14.0.1.Final/wildfly-15.0.1.Final/standalone/data/activemq/journal,bindingsDirectory=/opt/wildfly-14.0.1.Final/wildfly-15.0.1.Final/standalone/data/activemq/bindings,largeMessagesDirectory=/opt/wildfly-14.0.1.Final/wildfly-15.0.1.Final/standalone/data/activemq/largemessages,pagingDirectory=/opt/wildfly-14.0.1.Final/wildfly-15.0.1.Final/standalone/data/activemq/paging) 08:35:25,626 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221012: Using AIO Journal 08:35:26,433 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 238,813,184 08:35:26,610 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE 08:35:26,611 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP 08:35:26,611 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ 08:35:26,611 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP 08:35:26,740 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221034: Waiting indefinitely to obtain live lock 08:35:26,740 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221035: Live Server Obtained live lock 08:35:26,773 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-4) ISPN000128: Infinispan version: Infinispan 'Infinity Minus ONE +2' 9.4.3.Final 08:35:27,264 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000078: Starting JGroups channel ejb 08:35:27,276 INFO [org.infinispan.CLUSTER] (MSC service thread 1-4) ISPN000094: Received new cluster view for channel ejb: [test-1|1] (2) [test-1, test-2] 08:35:27,306 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000079: Channel ejb local address is test-2, physical addresses are [10.10.192.12:7600] 08:35:27,455 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000078: Starting JGroups channel ejb 08:35:27,460 INFO [org.infinispan.CLUSTER] (MSC service thread 1-4) ISPN000094: Received new cluster view for channel ejb: [test-1|1] (2) test-1, test-2] 08:35:27,490 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000079: Channel ejb local address is test-2, physical addresses are [10.10.192.12:7600] 08:35:27,611 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000078: Starting JGroups channel ejb 08:35:27,611 INFO [org.infinispan.CLUSTER] (MSC service thread 1-4) ISPN000094: Received new cluster view for channel ejb: [test-1|1] (2) test-1, test-2] 08:35:27,638 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000079: Channel ejb local address is test-2, physical addresses are [10.10.192.12:7600] 08:35:27,748 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000078: Starting JGroups channel ejb 08:35:27,749 INFO [org.infinispan.CLUSTER] (MSC service thread 1-4) ISPN000094: Received new cluster view for channel ejb: [test-1|1] (2) [test-1, test2] 08:35:27,810 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000079: Channel ejb local address is test-2, physical addresses are [10.10.192.12:7600] 08:35:27,980 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-4) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor-throughput acceptor 08:35:27,989 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-4) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor acceptor 08:35:27,989 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-4) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor-throughput acceptor 08:35:27,989 INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-4) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor acceptor ... 08:35:28,217 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221007: Server is now live 08:35:28,217 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.6.3.jbossorg-00014 [default, nodeID=81d46363-b774-11e9-8da2-0050568b6fed]
I'm using following configuration in standalone.xml:
...
<subsystem xmlns="urn:jboss:domain:jgroups:6.0">
<channels default="ee">
<channel name="ee" stack="tcp" cluster="ejb"/>
</channels>
<stacks>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<socket-protocol type="MPING" socket-binding="jgroups-mping"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
</stacks>
</subsystem>
...
<subsystem xmlns="urn:jboss:domain:messaging-activemq:5.0">
<server name="default">
<cluster password="${jboss.messaging.cluster.password:clusterpass}"/>
<security-setting name="#">
<role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
</security-setting>
<address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="0"/>
<address-setting name="jms.topic.GWConfiguration"/>
<http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
<http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
<param name="batch-delay" value="50"/>
</http-connector>
<in-vm-connector name="in-vm" server-id="0">
<param name="buffer-pooling" value="false"/>
</in-vm-connector>
<http-acceptor name="http-acceptor" http-listener="default"/>
<http-acceptor name="http-acceptor-throughput" http-listener="default">
<param name="batch-delay" value="50"/>
<param name="direct-deliver" value="false"/>
</http-acceptor>
<in-vm-acceptor name="in-vm" server-id="0">
<param name="buffer-pooling" value="false"/>
</in-vm-acceptor>
<broadcast-group name="bg-group1" jgroups-cluster="activemq-cluster" connectors="http-connector" />
<discovery-group name="dg-group1" jgroups-cluster="activemq-cluster"/>
<cluster-connection name="my-cluster" address="jms" connector-name="http-connector" message-load-balancing-type="ON_DEMAND" discovery-group="dg-group1"/>
<jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
<jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
<jms-topic name="GWConfiguration" entries="java:/jms/topic/GWConfiguration"/>
<connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
<connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" ha="true" block-on-acknowledge="true" reconnect-attempts="-1"/>
<pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" transaction="xa"/>
</server>
</subsystem>
...
<interfaces>
<interface name="management">
<inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
</interface>
<interface name="public">
<any-address/>
</interface>
<interface name="private">
<inet-address value="${jboss.bind.address.private:10.10.192.12}"/>
</interface>
<interface name="unsecure">
<inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>
</interface>
</interfaces>
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
<socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9999}"/>
<socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
<socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
<socket-binding name="http" port="${jboss.http.port:8080}">
<client-mapping destination-address="${jboss.host.name}"/>
</socket-binding>
<socket-binding name="https" port="${jboss.https.port:8443}"/>
<socket-binding name="txn-recovery-environment" port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
<socket-binding name="jgroups-mping" interface="private" multicast-address="${jboss.default.multicast.address:239.192.0.1}" multicast-port="45700"/>
<socket-binding name="jgroups-tcp" interface="private" port="7600"/>
<socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>
<socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:239.192.0.1}" multicast-port="45688"/>
<socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>
<socket-binding name="modcluster" multicast-address="${jboss.modcluster.multicast.address:239.192.0.1}" multicast-port="23364"/>
<outbound-socket-binding name="mail-smtp">
<remote-destination host="10.10.93.12" port="25"/>
</outbound-socket-binding>
</socket-binding-group>