0 Replies Latest reply on Aug 9, 2019 3:40 AM by Luboslav Mazanik

    (ActiveMQ) ClusterConnectionBridge not created in Wildfly 15 cluster (domain)

    Luboslav Mazanik Newbie

      Hello, I have a following issue with Wildfly 15 cluster (also tried with latest Wildfly 17).

       

      ClusterConnectionBridge is not being created after Wildfly is started, so messages are not being synchronized between servers in the cluster when they are submitted via topic.

       

      This is how it works with Wildfly 14 (log grep for "org.apache.activemq.artemis.core.server"):

      2019-08-05 15:32:56,057 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221012: Using AIO Journal
      2019-08-05 15:32:56,697 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 954,466,304
      2019-08-05 15:32:57,019 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
      2019-08-05 15:32:57,019 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
      2019-08-05 15:32:57,019 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
      2019-08-05 15:32:57,020 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
      2019-08-05 15:32:57,110 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221034: Waiting indefinitely to obtain live lock
      2019-08-05 15:32:57,110 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221035: Live Server Obtained live lock
      2019-08-05 15:32:58,349 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221007: Server is now live
      2019-08-05 15:32:58,350 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 83) AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.6.3.jbossorg-001 [default, nodeID=ceb7187b-fd8a-11e8-8274-0050568b6fed]
      2019-08-05 15:32:59,410 WARN  [org.apache.activemq.artemis.core.server] (Thread-2 (ActiveMQ-client-global-threads)) AMQ224078: The size of duplicate cache detection (<id_cache-size>) appears to be too large 20,000. It should be no greater than the number of messages that can be squeezed into conformation buffer () 1,048,576.
      2019-08-05 15:32:59,779 INFO  [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1426f525)) AMQ221027: Bridge ClusterConnectionBridge@484e51c8 [name=$.artemis.internal.sf.my-cluster.f381572d-fde8-11e8-8ee3-0050568b484f, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.f381572d-fde8-11e8-8ee3-0050568b484f, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=ceb7187b-fd8a-11e8-8274-0050568b6fed], temp=false]@26eea0e targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@484e51c8 [name=$.artemis.internal.sf.my-cluster.f381572d-fde8-11e8-8ee3-0050568b484f, queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.f381572d-fde8-11e8-8ee3-0050568b484f, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=ceb7187b-fd8a-11e8-8274-0050568b6fed], temp=false]@26eea0e targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=test-2], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@75434150[nodeUUID=ceb7187b-fd8a-11e8-8274-0050568b6fed, connector=TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=test-1, address=jms, server=ActiveMQServerImpl::serverUUID=ceb7187b-fd8a-11e8-8274-0050568b6fed])) [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=test-2], discoveryGroupConfiguration=null]] is connected

       

      In Wildfly 15 (using same configuration and same application builds), cluster is well formed, ActiveMQ broker is started, but log message with ClusterConnectionBridge is never logged. Any message submitted to a specific topic is NOT broadcasted, only is "received" and processed on a server where it is submitted.

      ...
      08:35:25,535 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/opt/wildfly-14.0.1.Final/wildfly-15.0.1.Final/standalone/data/activemq/journal,bindingsDirectory=/opt/wildfly-14.0.1.Final/wildfly-15.0.1.Final/standalone/data/activemq/bindings,largeMessagesDirectory=/opt/wildfly-14.0.1.Final/wildfly-15.0.1.Final/standalone/data/activemq/largemessages,pagingDirectory=/opt/wildfly-14.0.1.Final/wildfly-15.0.1.Final/standalone/data/activemq/paging)
      08:35:25,626 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221012: Using AIO Journal
      08:35:26,433 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 238,813,184
      08:35:26,610 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
      08:35:26,611 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
      08:35:26,611 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
      08:35:26,611 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
      08:35:26,740 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221034: Waiting indefinitely to obtain live lock
      08:35:26,740 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221035: Live Server Obtained live lock
      08:35:26,773 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-4) ISPN000128: Infinispan version: Infinispan 'Infinity Minus ONE +2' 9.4.3.Final
      08:35:27,264 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000078: Starting JGroups channel ejb
      08:35:27,276 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-4) ISPN000094: Received new cluster view for channel ejb: [test-1|1] (2) [test-1, test-2]
      08:35:27,306 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000079: Channel ejb local address is test-2, physical addresses are [10.10.192.12:7600]
      08:35:27,455 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000078: Starting JGroups channel ejb
      08:35:27,460 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-4) ISPN000094: Received new cluster view for channel ejb: [test-1|1] (2) test-1, test-2]
      08:35:27,490 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000079: Channel ejb local address is test-2, physical addresses are [10.10.192.12:7600]
      08:35:27,611 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000078: Starting JGroups channel ejb
      08:35:27,611 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-4) ISPN000094: Received new cluster view for channel ejb: [test-1|1] (2) test-1, test-2]
      08:35:27,638 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000079: Channel ejb local address is test-2, physical addresses are [10.10.192.12:7600]
      08:35:27,748 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000078: Starting JGroups channel ejb
      08:35:27,749 INFO  [org.infinispan.CLUSTER] (MSC service thread 1-4) ISPN000094: Received new cluster view for channel ejb: [test-1|1] (2) [test-1, test2]
      08:35:27,810 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000079: Channel ejb local address is test-2, physical addresses are [10.10.192.12:7600]
      08:35:27,980 INFO  [org.wildfly.extension.messaging-activemq] (MSC service thread 1-4) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor-throughput acceptor
      08:35:27,989 INFO  [org.wildfly.extension.messaging-activemq] (MSC service thread 1-4) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor acceptor
      08:35:27,989 INFO  [org.wildfly.extension.messaging-activemq] (MSC service thread 1-4) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor-throughput acceptor
      08:35:27,989 INFO  [org.wildfly.extension.messaging-activemq] (MSC service thread 1-4) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor acceptor
      ...
      08:35:28,217 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221007: Server is now live
      08:35:28,217 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 80) AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.6.3.jbossorg-00014 [default, nodeID=81d46363-b774-11e9-8da2-0050568b6fed]

       

      I'm using following configuration in standalone.xml:

      ...

      <subsystem xmlns="urn:jboss:domain:jgroups:6.0">

                  <channels default="ee">

                      <channel name="ee" stack="tcp" cluster="ejb"/>

                  </channels>

                  <stacks>

                      <stack name="tcp">

                          <transport type="TCP" socket-binding="jgroups-tcp"/>

                          <socket-protocol type="MPING" socket-binding="jgroups-mping"/>

                          <protocol type="MERGE3"/>

                          <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                          <protocol type="FD_ALL"/>

                          <protocol type="VERIFY_SUSPECT"/>

                          <protocol type="pbcast.NAKACK2"/>

                          <protocol type="UNICAST3"/>

                          <protocol type="pbcast.STABLE"/>

                          <protocol type="pbcast.GMS"/>

                          <protocol type="MFC"/>

                          <protocol type="FRAG3"/>

                      </stack>

                  </stacks>

              </subsystem>

      ...

              <subsystem xmlns="urn:jboss:domain:messaging-activemq:5.0">

                  <server name="default">

                      <cluster password="${jboss.messaging.cluster.password:clusterpass}"/>

                      <security-setting name="#">

                          <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>

                      </security-setting>

                      <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="0"/>

                      <address-setting name="jms.topic.GWConfiguration"/>

                      <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>

                      <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">

                          <param name="batch-delay" value="50"/>

                      </http-connector>

                      <in-vm-connector name="in-vm" server-id="0">

                          <param name="buffer-pooling" value="false"/>

                      </in-vm-connector>

                      <http-acceptor name="http-acceptor" http-listener="default"/>

                      <http-acceptor name="http-acceptor-throughput" http-listener="default">

                          <param name="batch-delay" value="50"/>

                          <param name="direct-deliver" value="false"/>

                      </http-acceptor>

                      <in-vm-acceptor name="in-vm" server-id="0">

                          <param name="buffer-pooling" value="false"/>

                      </in-vm-acceptor>

                      <broadcast-group name="bg-group1" jgroups-cluster="activemq-cluster" connectors="http-connector" />

                      <discovery-group name="dg-group1" jgroups-cluster="activemq-cluster"/>

                      <cluster-connection name="my-cluster" address="jms" connector-name="http-connector" message-load-balancing-type="ON_DEMAND" discovery-group="dg-group1"/>

                      <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>

                      <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>

                      <jms-topic name="GWConfiguration" entries="java:/jms/topic/GWConfiguration"/>

                      <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>

                      <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" ha="true" block-on-acknowledge="true" reconnect-attempts="-1"/>

                      <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" transaction="xa"/>

                  </server>

              </subsystem>

      ...

          <interfaces>

              <interface name="management">

                  <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>

              </interface>

              <interface name="public">

                  <any-address/>

              </interface>

              <interface name="private">

                  <inet-address value="${jboss.bind.address.private:10.10.192.12}"/>

              </interface>

              <interface name="unsecure">

                  <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>

              </interface>

          </interfaces>

          <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">

              <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9999}"/>

              <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>

              <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>

              <socket-binding name="http" port="${jboss.http.port:8080}">

                  <client-mapping destination-address="${jboss.host.name}"/>

              </socket-binding>

              <socket-binding name="https" port="${jboss.https.port:8443}"/>

              <socket-binding name="txn-recovery-environment" port="4712"/>

              <socket-binding name="txn-status-manager" port="4713"/>

              <socket-binding name="jgroups-mping" interface="private" multicast-address="${jboss.default.multicast.address:239.192.0.1}" multicast-port="45700"/>

              <socket-binding name="jgroups-tcp" interface="private" port="7600"/>

              <socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>

              <socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:239.192.0.1}" multicast-port="45688"/>

              <socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>

              <socket-binding name="modcluster" multicast-address="${jboss.modcluster.multicast.address:239.192.0.1}" multicast-port="23364"/>

              <outbound-socket-binding name="mail-smtp">

                  <remote-destination host="10.10.93.12" port="25"/>

              </outbound-socket-binding>

          </socket-binding-group>