5 Replies Latest reply on Jun 16, 2017 4:11 PM by jbertram

    Why HornetQConnectionFactory client does not automatic failover?

    alireza.alallah
    alireza.alallah Jun 14, 2017 2:05 AM

      I have clustered two wildfly9 servers, i want to use load-balancing and failover JMS feature in domain mode,config  as below:

       

      master ip: 192.168.1.1 

      slave ip**:  192.168.1.2 

      hornetq ver 2.4.7

      wildfly 9

       

       

      During while loop(send dummy msg) failover occure on master , client connectiion  does switch to slave and thrown exception,why??,I spending many times to solve this problem, still not solve this problem

       

       

          <hornetq-server>

                              <clustered>true</clustered>

                              <cluster-password>my-pass</cluster-password>

                              <allow-failback>true</allow-failback>

                              <failover-on-shutdown>true</failover-on-shutdown>

                              <shared-store>false</shared-store>

                              <journal-file-size>102400</journal-file-size>

         

                              <connectors>

                                  <netty-connector name="netty" socket-binding="messaging"/>

                                  <netty-connector name="netty-throughput" socket-binding="messaging-throughput">

                                      <param key="batch-delay" value="50"/>

                                  </netty-connector>

                                  <in-vm-connector name="in-vm" server-id="0"/>

                                  <connector name="node1">

                                      <param key="host" value="192.168.1.2"/>

                                      <param key="port" value="5445"/>

                                      <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                                  </connector>

                              </connectors>

          <acceptors>

                                  <netty-acceptor name="netty" socket-binding="messaging"/>

                                  <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">

                                      <param key="batch-delay" value="50"/>

                                      <param key="direct-deliver" value="false"/>

                                  </netty-acceptor>

                                  <in-vm-acceptor name="in-vm" server-id="0"/>

                              </acceptors>

         

                              <cluster-connections>

                                  <cluster-connection name="my-cluster">

                                      <address>jms</address>

                                      <connector-ref>netty</connector-ref>

                                      <check-period>1000</check-period>

                                      <connection-ttl>1000</connection-ttl>

                                      <reconnect-attempts>3</reconnect-attempts>

                                      <static-connectors>

                                          <connector-ref>

                                              node1

                                          </connector-ref>

                                      </static-connectors>

                                  </cluster-connection>

                              </cluster-connections>

         

                              <security-settings>

                                  <security-setting match="#">

                                      <permission type="send" roles="guest"/>

                                      <permission type="consume" roles="guest"/>

                                      <permission type="createNonDurableQueue" roles="guest"/>

                                      <permission type="deleteNonDurableQueue" roles="guest"/>

                                  </security-setting>

                              </security-settings>

         

          <address-settings>

                                  <address-setting match="#">

                                      <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                                      <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                                      <max-size-bytes>10485760</max-size-bytes>

                                      <page-size-bytes>2097152</page-size-bytes>

                                      <message-counter-history-day-limit>10</message-counter-history-day-limit>

                                      <redistribution-delay>1000</redistribution-delay>

                                  </address-setting>

                              </address-settings>

         

                              <jms-connection-factories>

                                  <connection-factory name="InVmConnectionFactory">

                                      <connectors>

                                          <connector-ref connector-name="in-vm"/>

                                      </connectors>

                                      <entries>

                                          <entry name="java:/ConnectionFactory"/>

                                      </entries>

                                  </connection-factory>

                                  <connection-factory name="RemoteConnectionFactory">

                                      <connectors>

                                          <connector-ref connector-name="netty"/>

                                      </connectors>

                                      <entries>

                                          <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>

                                      </entries>

                                      <ha>true</ha>

          <block-on-durable-send>true</block-on-durable-send>

          <block-on-acknowledge>true</block-on-acknowledge>

          <connection-ttl>600000</connection-ttl>

          <call-timeout>180000</call-timeout>

          <client-failure-check-period>60000</client-failure-check-period>

          <block-on-non-durable-send>true</block-on-non-durable-send>

          <retry-interval>4000</retry-interval>

          <reconnect-attempts>5</reconnect-attempts>

          <confirmation-window-size>1000000</confirmation-window-size>

          <consumer-window-size>3145728</consumer-window-size>

         

                                  </connection-factory>

                                  <pooled-connection-factory name="hornetq-ra">

                                      <transaction mode="xa"/>

                                      <connectors>

                                          <connector-ref connector-name="in-vm"/>

                                          <connector-ref connector-name="node1"/>

                                      </connectors>

                                      <entries>

                                          <entry name="java:/JmsXA"/>

                                          <entry name="java:jboss/DefaultJMSConnectionFactory"/>

                                      </entries>

                                      <ha>true</ha>

                                      <block-on-acknowledge>true</block-on-acknowledge>

                                      <reconnect-attempts>-1</reconnect-attempts>

                                  </pooled-connection-factory>

                              </jms-connection-factories>

       

       

      slave configuration as below:

       

       

           <hornetq-server>

                              <clustered>true</clustered>

                              <cluster-password>my-pass</cluster-password>

                              <shared-store>false</shared-store>

                              <journal-file-size>102400</journal-file-size>

                              <allow-failback>true</allow-failback>

                              <failover-on-shutdown>true</failover-on-shutdown>

                              <backup>true</backup>

                              <connectors>

                                 <netty-connector name="netty" socket-binding="messaging"/>

                                 <netty-connector name="netty-throughput" socket-binding="messaging-throughput">

                                   <param key="batch-delay" value="50"/>

                                  </netty-connector>

                                  <in-vm-connector name="in-vm" server-id="1"/>

                                  <connector name="node-master">

                                     <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                                     <param key="host" value="192.168.1.1"/>

                                    <param key="port" value="5445"/>

                                  </connector>

                              </connectors>

          <acceptors>

                                 <netty-acceptor name="netty" socket-binding="messaging"/>

                                 <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">

                                   <param key="batch-delay" value="50"/>

                                   <param key="direct-deliver" value="false"/>

                                 </netty-acceptor>

                                 <in-vm-acceptor name="in-vm" server-id="1"/>

                               </acceptors>

         

                              <cluster-connections>

                                  <cluster-connection name="my-cluster">

                                      <address>jms</address>

                                      <connector-ref>netty</connector-ref>

                                      <reconnect-attempts>3</reconnect-attempts>

                                      <connection-ttl>1000</connection-ttl>

                                      <check-period>1000</check-period>

                                      <static-connectors>

                                          <connector-ref>node-master</connector-ref>

                                      </static-connectors>

                                  </cluster-connection>

                              </cluster-connections>

                              <security-settings>

                                  <security-setting match="#">

                                      <permission type="send" roles="guest"/>

                                      <permission type="consume" roles="guest"/>

                                      <permission type="createNonDurableQueue" roles="guest"/>

                                      <permission type="deleteNonDurableQueue" roles="guest"/>

                                  </security-setting>

                              </security-settings>

         

           </address-settings>

                              <jms-connection-factories>

                                  <connection-factory name="InVmConnectionFactory">

                                      <connectors>

                                          <connector-ref connector-name="in-vm"/>

                                      </connectors>

                                      <entries>

                                          <entry name="java:/ConnectionFactory"/>

                                      </entries>

                                  </connection-factory>

                                  <connection-factory name="RemoteConnectionFactory">

                                      <connectors>

                                          <connector-ref connector-name="netty"/>

                                      </connectors>

                                      <entries>

                                          <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>

                                      </entries>

                                      <block-on-acknowledge>true</block-on-acknowledge>

                                       <ha>true</ha>

          <block-on-durable-send>true</block-on-durable-send>

          <block-on-acknowledge>true</block-on-acknowledge>

          <connection-ttl>600000</connection-ttl>

          <call-timeout>180000</call-timeout>

          <client-failure-check-period>60000</client-failure-check-period>

          <block-on-non-durable-send>true</block-on-non-durable-send>

          <retry-interval>4000</retry-interval>

          <reconnect-attempts>5</reconnect-attempts>

          <confirmation-window-size>1000000</confirmation-window-size>

          <consumer-window-size>3145728</consumer-window-size>

         

                                  </connection-factory>

         

           <pooled-connection-factory name="hornetq-ra">

                                      <transaction mode="xa"/>

                                      <connectors>

                                          <connector-ref connector-name="in-vm"/>

                                          <connector-ref connector-name="node-master"/>

                                      </connectors>

                                      <entries>

                                          <entry name="java:/JmsXA"/>

                                          <!-- Global JNDI entry used to provide a default JMS Connection factory to EE application -->

                                          <entry name="java:jboss/DefaultJMSConnectionFactory"/>

                                      </entries>

                                      <ha>true</ha>

                                      <block-on-acknowledge>true</block-on-acknowledge>

                                      <reconnect-attempts>-1</reconnect-attempts>

                                  </pooled-connection-factory>

                              </jms-connection-factories>

       

       

      my client code as below:

       

       

           Map<String, Object> connectionParams = new HashMap<String, Object>();

                  connectionParams.put(TransportConstants.PORT_PROP_NAME, "port+port-offset");

                  connectionParams.put(TransportConstants.HOST_PROP_NAME, "192.168.1.1");

         

         

                  Map<String, Object> connectionParams2 = new HashMap<String, Object>();

                  connectionParams2.put(TransportConstants.PORT_PROP_NAME, "port+port-offset");

                  connectionParams2.put(TransportConstants.HOST_PROP_NAME, "192.168.1.2");

         

         

                  TransportConfiguration transportConfiguration = new TransportConfiguration(NettyConnectorFactory.class.getName(), connectionParams);

                  TransportConfiguration transportConfiguration1 = new TransportConfiguration(NettyConnectorFactory.class.getName(), connectionParams2);

                  //

         

         

         

                  HornetQConnectionFactory connectionFactory = HornetQJMSClient.createConnectionFactoryWithHA(JMSFactoryType.CF, transportConfiguration1, transportConfiguration);

         

         

                  connectionFactory.setClientFailureCheckPeriod(1000);

                  connectionFactory.setReconnectAttempts(3);

                  connectionFactory.setConnectionTTL(1000);

         

         

         

                  Connection connection = null;

                  try {

                      connection = connectionFactory.createConnection("username", "password");

                  } catch (JMSException e) {

                      // TODO Auto-generated catch block

                      e.printStackTrace();

                  }

                  if (connection != null) {

                      System.out.println("JMS Connection succeed.");

         

         

                  }

                  // Step 9. create the JMS management queue.

                  // It is a "special" queue and it is not looked up from JNDI but

                  // constructed directly

                  // see hornetq-jms.xml configuration file in server side

                  Queue queue = HornetQJMSClient.createQueue("myQueue");

         

                  Session session = null;

                  try {

                      session = connection.createSession(false, HornetQJMSConstants.PRE_ACKNOWLEDGE);

                  } catch (JMSException e) {

                      e.printStackTrace();

                  }

         

         

                  MessageProducer producer = null;

                  try {

                      producer = session.createProducer(queue);

                  } catch (JMSException e) {

                      e.printStackTrace();

                  }

                 

                  while (true) {

                      try {

                          producer.send(session.createTextMessage("asdasdas"));

                      } catch (Exception e) {

                          e.printStackTrace();

                      }

                  }

       

      when master shutdown (with ctrl+c) client thrown exception (Session Closed)

       

      When Master/Slave Shutdown , client does not failover to another server, how top solve this problem?

      Thanks in advanced

        • 1. Re: Why HornetQConnectionFactory client does not automatic failover?
          jbertram
          jbertram Jun 14, 2017 10:01 AM (in response to alireza.alallah)

          Can you post the logs from both the live and the backup servers?

            • Actions
          • 2. Re: Why HornetQConnectionFactory client does not automatic failover?
            alireza.alallah
            alireza.alallah Jun 15, 2017 10:50 AM (in response to jbertram)

            Hi Justin, thanks for reply

            I have migrated to wildfly10 and artemisq , but i have same problem, my client exception is

             

            Jun 15, 2017 6:42:05 PM org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl fail

            WARN: AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]

            Failover event triggered :FAILURE_DETECTED

            Jun 15, 2017 6:42:09 PM org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl getConnectionWithRetry

            WARN: AMQ212005: Tried 5 times to connect. Now giving up on reconnecting it.

            Failover event triggered :FAILOVER_FAILED

            Exception in thread "main" javax.jms.JMSException: AMQ119016: Connection failure detected. Unblocking a blocking call that will never get a response

              at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:406)

              at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:304)

              at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:378)

              at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:286)

              at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:264)

              at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:122)

              at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:477)

              at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:173)

              at Application.main(Application.java:65)

            Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED message=AMQ119016: Connection failure detected. Unblocking a blocking call that will never get a response]

              ... 9 more

            my master log (Stop with Ctrl+C):

            Server:server-three] 10:26:57,901 DEBUG [org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl] (ServerService Thread Pool -- 76) Message ServerMessage[messageID=1486,durable=true,userID=null,priority=0, bodySize=512, timestamp=0,expiration=0, durable=true, address=activemq.notifications,properties=TypedProperties[factory=org.apache.activemq.artemis.core.remoting.impl.invm.InVMAcceptorFactory,_AMQ_NotifType=ACCEPTOR_STOPPED,_AMQ_NotifTimestamp=1497536817901,id=0]]@1041289119 is not going anywhere as it didn't have a binding on address:activemq.notifications

            [Server:server-three] 10:26:57,916 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 76) AMQ221002: Apache ActiveMQ Artemis Message Broker version 1.1.0.wildfly-017 [4bfaac95-51d6-11e7-9a67-39317e17614a] stopped

            [Server:server-three] 10:26:57,917 DEBUG [org.jboss.as.security] (MSC service thread 1-1) Stopping security domain service other

            [Server:server-three] 10:26:57,917 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0008: Undertow HTTP listener default suspending

            [Server:server-three] 10:26:57,918 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 192.168.0.200:8330

            [Server:server-three] 10:26:57,925 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0004: Undertow 1.4.0.Final stopping

            [Server:server-three] 10:26:57,930 DEBUG [org.jgroups.protocols.UDP] (multicast receiver,ee,master:server-three) receiver socket is closed, exception=java.net.SocketException: Socket closed

            [Server:server-three] 10:26:57,930 DEBUG [org.jgroups.protocols.UDP] (multicast receiver,ee,master:server-three) multicast receiver thread terminated

            [Server:server-three] 10:26:57,930 DEBUG [org.jgroups.protocols.UDP] (MSC service thread 1-2) multicast socket closed

            [Server:server-three] 10:26:57,931 DEBUG [org.jgroups.protocols.UDP] (unicast receiver,ee,master:server-three) receiver socket is closed, exception=java.net.SocketException: Socket closed

            [Server:server-three] 10:26:57,931 DEBUG [org.jgroups.protocols.UDP] (unicast receiver,ee,master:server-three) unicast receiver thread terminated

            [Server:server-three] 10:26:57,931 DEBUG [org.jgroups.protocols.UDP] (MSC service thread 1-2) closing sockets and stopping threads

            [Server:server-three] 10:26:57,950 DEBUG [org.jboss.as.server] (ServerService Thread Pool -- 76) DeploymentMountProvider stopped

            [Host Controller] 10:26:57,955 INFO  [org.jboss.as.host.controller] (management I/O-1) WFLYHC0027: Unregistering server server-three

            [Server:server-three] 10:26:57,970 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: WildFly Full 10.1.0.Final (WildFly Core 2.2.0.Final) stopped in 309ms

            [Server:server-three]

            10:26:58,296 INFO  [org.jboss.as.process.Server:server-three.status] (reaper for Server:server-three) WFLYPC0011: Process 'Server:server-three' finished with an exit status of 130

            [Host Controller] 10:26:58,298 INFO  [org.jboss.as.host.controller] (ProcessControllerConnection-thread - 2) WFLYHC0027: Unregistering server server-three

            [Host Controller] 10:26:58,339 WARN  [org.jboss.as.domain.controller] (MSC service thread 1-1) WFLYHC0030: Connection to remote host "slave" closed unexpectedly

            [Host Controller] 10:26:58,351 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: WildFly Full 10.1.0.Final (WildFly Core 2.2.0.Final) stopped in 723ms

            [Host Controller]

            10:26:58,356 INFO  [org.jboss.as.process.Host Controller.status] (reaper for Host Controller) WFLYPC0011: Process 'Host Controller' finished with an exit status of 130

            10:26:58,358 INFO  [org.jboss.as.process] (Shutdown thread) WFLYPC0016: All processes finished; exiting

             

            and my slave log:

            [Server:server-three] 10:26:50,666 WARN  [org.jgroups.protocols.UDP] (thread-1,ee,slave:server-three) JGRP000012: discarded message from different cluster activemq-cluster (our cluster is ee). Sender was slave:server-three (received 72 identical messages from slave:server-three in the last 60025 ms)

            [Server:server-three] 10:26:58,033 DEBUG [org.jboss.activemq.artemis.wildfly.integration.recovery] (Periodic Recovery)

            [Server:server-three] =======================================================================================

            [Server:server-three] 10:26:58,033 DEBUG [org.jboss.activemq.artemis.wildfly.integration.recovery] (Periodic Recovery) Returning the following list on getXAREsources:

            [Server:server-three] 10:26:58,033 DEBUG [org.jboss.activemq.artemis.wildfly.integration.recovery] (Periodic Recovery) server-id=944907bb-51d4-11e7-b51d-43814264a759, value=org.jboss.activemq.artemis.wildfly.integration.WildFlyActiveMQXAResourceWrapper@6a8cc10c

            [Server:server-three] 10:26:58,033 DEBUG [org.jboss.activemq.artemis.wildfly.integration.recovery] (Periodic Recovery) =======================================================================================

            [Server:server-three]

            [Server:server-three] 10:26:58,033 DEBUG [org.jboss.ejb.client.txn] (Periodic Recovery) Send recover request for transaction origin node identifier 1 to EJB receiver with node name slave:server-three

            [Server:server-three] 10:26:58,033 DEBUG [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) looking for recover at ClientSessionImpl [name=df549be0-51d4-11e7-b51d-43814264a759, username=null, closed=false, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@10a82f1, metaData=()]@5613459d configuration [XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA]]

            [Host Controller] 10:26:58,322 WARN  [org.jboss.as.host.controller] (management task-1) WFLYHC0015: Connection to remote host-controller closed. Trying to reconnect.

            [Host Controller] 10:26:58,324 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 10) WFLYHC0150: Trying to reconnect to master host controller.

            [Host Controller] 10:26:59,330 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 10) WFLYHC0150: Trying to reconnect to master host controller.

            [Server:server-three] 10:27:02,580 DEBUG [org.apache.activemq.artemis.core.server.impl.QueueImpl] (activemq-expiry-reaper-thread) Cannot expire from jms.queue.ExpiryQueue into jms.queue.ExpiryQueue

            [Host Controller] 10:27:03,332 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 10) WFLYHC0150: Trying to reconnect to master host controller.

            [Server:server-three] 10:27:04,392 DEBUG [org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl] (Thread-1 (ActiveMQ-client-netty-threads-264441192)) Invocation of interceptor org.apache.activemq.artemis.core.server.impl.ReplicationError on PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=60000] returned true

            [Server:server-three] 10:27:08,037 DEBUG [org.jboss.ejb.client.txn] (Periodic Recovery) Send recover request for transaction origin node identifier 1 to EJB receiver with node name slave:server-three

            [Server:server-three] 10:27:08,037 DEBUG [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) looking for recover at ClientSessionImpl [name=df549be0-51d4-11e7-b51d-43814264a759, username=null, closed=false, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@10a82f1, metaData=()]@5613459d configuration [XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA]]

            [Host Controller] 10:27:12,335 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 10) WFLYHC0150: Trying to reconnect to master host controller.

            [Host Controller] 10:27:27,345 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 10) WFLYHC0150: Trying to reconnect to master host controller.

            [Server:server-three] 10:27:32,581 DEBUG [org.apache.activemq.artemis.core.server.impl.QueueImpl] (activemq-expiry-reaper-thread) Cannot expire from jms.queue.ExpiryQueue into jms.queue.ExpiryQueue

            [Server:server-three] 10:27:34,393 DEBUG [org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl] (Thread-1 (ActiveMQ-client-netty-threads-264441192)) Invocation of interceptor org.apache.activemq.artemis.core.server.impl.ReplicationError on PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=60000] returned true

            [Server:server-three] 10:27:41,549 WARN  [org.jgroups.protocols.UDP] (thread-1,activemq-cluster,slave:server-three) JGRP000012: discarded message from different cluster ee (our cluster is activemq-cluster). Sender was slave:server-three (received 14 identical messages from slave:server-three in the last 60169 ms)

            [Host Controller] 10:27:42,347 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 10) WFLYHC0150: Trying to reconnect to master host controller.

            [Server:server-three] 10:27:50,691 WARN  [org.jgroups.protocols.UDP] (thread-2,ee,slave:server-three) JGRP000012: discarded message from different cluster activemq-cluster (our cluster is ee). Sender was slave:server-three (received 74 identical messages from slave:server-three in the last 60025 ms)

            [Host Controller] 10:27:57,350 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 10) WFLYHC0150: Trying to reconnect to master host controller.

            [Server:server-three] 10:28:02,581 DEBUG [org.apache.activemq.artemis.core.server.impl.QueueImpl] (activemq-expiry-reaper-thread) Cannot expire from jms.queue.ExpiryQueue into jms.queue.ExpiryQueue

            [Server:server-three] 10:28:04,393 DEBUG [org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl] (Thread-1 (ActiveMQ-client-netty-threads-264441192)) Invocation of interceptor org.apache.activemq.artemis.core.server.impl.ReplicationError on PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=60000] returned true

            [Host Controller] 10:28:12,354 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 10) WFLYHC0150: Trying to reconnect to master host controller.

            [Host Controller] 10:28:27,357 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 10) WFLYHC0150: Trying to reconnect to master host controller.

            [Server:server-three] 10:28:32,582 DEBUG [org.apache.activemq.artemis.core.server.impl.QueueImpl] (activemq-expiry-reaper-thread) Cannot expire from jms.queue.ExpiryQueue into jms.queue.ExpiryQueue

            [Server:server-three] 10:28:34,394 DEBUG [org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl] (Thread-1 (ActiveMQ-client-netty-threads-264441192)) Invocation of interceptor org.apache.activemq.artemis.core.server.impl.ReplicationError on PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=60000] returned true

             

            my master config:

            <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">

                            <server name="default">

                                <security enabled="false"/>

                                <cluster password="${jboss.messaging.cluster.password:sh@1396SM}"/>

             

             

                                <replication-master check-for-live-server="true" group-name="ali-server-group" cluster-name="my-cluster"/>

                                <security-setting name="#">

                                    <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>

                                </security-setting>

                                <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>

                                <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>

                                <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">

                                    <param name="batch-delay" value="50"/>

                                </http-connector>

                                <in-vm-connector name="in-vm" server-id="0"/>

                                <http-acceptor name="http-acceptor" http-listener="default"/>

                                <http-acceptor name="http-acceptor-throughput" http-listener="default">

                                    <param name="batch-delay" value="50"/>

                                    <param name="direct-deliver" value="false"/>

                                </http-acceptor>

                                <in-vm-acceptor name="in-vm" server-id="0"/>

                                <broadcast-group name="bg-group1" jgroups-channel="activemq-cluster" connectors="http-connector"/>

                                <discovery-group name="dg-group1" jgroups-channel="activemq-cluster"/>

                                <cluster-connection name="my-cluster" address="jms" connector-name="http-connector" discovery-group="dg-group1"/>

                                <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>

                                <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>

                                <jms-queue name="ali" entries="java:jboss/exported/jms/queue/ali"/>

                                <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>

                                <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" retry-interval-multiplier="1.0" retry-interval="1000" client-failure-check-period="1000" failover-on-initial-connection="true"  connectors="http-connector" ha="true" block-on-acknowledge="true" reconnect-attempts="5"/>

                                <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" transaction="xa" reconnect-attempts="-1" block-on-acknowledge="true" ha="true" />

                            </server>

                            <server name="backup">

                            <security enabled="false"/>

                            <cluster password="sh@1396SM"/>

                            <replication-slave max-saved-replicated-journal-size="-1" group-name="group2" cluster-name="my-cluster">

                            <scale-down enabled="false"/>

                            </replication-slave>

                            <address-setting name="#" redistribution-delay="0" page-size-bytes="524288" max-size-bytes="1048576" max-delivery-attempts="200"/>

                            <remote-connector name="netty-backup" socket-binding="messaging-backup"/>

                            <in-vm-connector name="in-vm" server-id="0"/>

                            <remote-acceptor name="netty-backup" socket-binding="messaging-backup"/>

                            <broadcast-group name="bg-group-backup" connectors="netty-backup" broadcast-period="1000" jgroups-channel="activemq-cluster" jgroups-stack="udp"/>

                            <discovery-group name="dg-group-backup" refresh-timeout="1000" jgroups-channel="activemq-cluster" jgroups-stack="udp"/>

                            <cluster-connection name="my-cluster" discovery-group="dg-group-backup" retry-interval="1000" connector-name="netty-backup" address="jms"/>

                            </server>

                        </subsystem>

             

            my slave config:

             

             

            <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">

                            <server name="default">

                                <security enabled="false"/>

                                <cluster password="${jboss.messaging.cluster.password:sh@1396SM}"/>

                                <security-setting name="#">

                                    <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>

                                </security-setting>

                                <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>

                                <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>

                                <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">

                                    <param name="batch-delay" value="50"/>

                                </http-connector>

                                </connector>

                                <replication-master check-for-live-server="true" group-name="ali-server-group" cluster-name="my-cluster"/>

                                <in-vm-connector name="in-vm" server-id="0"/>

                                <http-acceptor name="http-acceptor" http-listener="default"/>

                                <http-acceptor name="http-acceptor-throughput" http-listener="default">

                                    <param name="batch-delay" value="50"/>

                                    <param name="direct-deliver" value="false"/>

                                </http-acceptor>

                                <remote-connector name="netty-backup" socket-binding="messaging-backup"/>

                                <remote-acceptor name="netty-backup" socket-binding="messaging-backup"/>

                                <in-vm-acceptor name="in-vm" server-id="0"/>

                                <broadcast-group name="bg-group1" jgroups-channel="activemq-cluster" connectors="http-connector"/>

                                <discovery-group name="dg-group1" jgroups-channel="activemq-cluster"/>

                                <cluster-connection name="my-cluster" address="jms" connector-name="http-connector" discovery-group="dg-group1" failover-on-initial-connection="true"/>

                                <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>

                                <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>

                                <jms-queue name="ali" entries="java:jboss/exported/jms/queue/ali"/>

                                <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>

                                <connection-factory name="RemoteConnectionFactory"  entries="java:jboss/exported/jms/RemoteConnectionFactory" failover-on-initial-connection="true" client-failure-check-period="1000" connectors="http-connector"  ha="true" retry-interval-multiplier="1.0" retry-interval="1000" block-on-acknowledge="true" reconnect-attempts="5"/>

                                <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" reconnect-attempts="-1" block-on-acknowledge="true" ha="true" connectors="in-vm" transaction="xa" block-on-acknowledge="true" />

                            </server>

                            <server name="backup">

                            <security enabled="false"/>

                            <cluster password="sh@1396SM"/>

                            <replication-slave max-saved-replicated-journal-size="-1" group-name="group2" cluster-name="my-cluster">

                            <scale-down enabled="false"/>

                            </replication-slave>

                            <address-setting name="#" redistribution-delay="0" page-size-bytes="524288" max-size-bytes="1048576" max-delivery-attempts="200"/>

                            <remote-connector name="netty-backup" socket-binding="messaging-backup"/>

                            <in-vm-connector name="in-vm" server-id="0"/>

                            <remote-acceptor name="netty-backup" socket-binding="messaging-backup"/>

                            <broadcast-group name="bg-group-backup" connectors="netty-backup" broadcast-period="1000" jgroups-channel="activemq-cluster" jgroups-stack="udp"/>

                            <discovery-group name="dg-group-backup" refresh-timeout="1000" jgroups-channel="activemq-cluster" jgroups-stack="udp"/>

                            <cluster-connection name="my-cluster" discovery-group="dg-group-backup" retry-interval="1000" connector-name="netty-backup" address="jms"/>

                            </server>

                        </subsystem>

             

            my client code:

            InitialContext initialContext = null;

               Connection connectionA = null;

              try {

               final Properties props = new Properties();
               props.put("java.naming.factory.initial",
               "org.jboss.naming.remote.client.InitialContextFactory");
               props.put("java.naming.provider.url",
               "http-remoting://192.168.0.200:8330,http-remoting://192.168.0.201:8330");


               props.put(Context.SECURITY_PRINCIPAL, "isc");
               // password
               props.put(Context.SECURITY_CREDENTIALS, "sh@1396SM");

               // Step 1. Get an initial context for looking up JNDI from server 0
               initialContext = new InitialContext(props);

               // Step 2. Look-up the JMS Queue object from JNDI

              // Step 3. Look-up a JMS Connection Factory object from JNDI on server 0
               ConnectionFactory connectionFactory = (ConnectionFactory) initialContext.lookup("jms/RemoteConnectionFactory");



               ((ActiveMQConnectionFactory)connectionFactory).setInitialConnectAttempts(20);
               ((ActiveMQConnectionFactory)connectionFactory).setFailoverOnInitialConnection(true);
               // Step 4. We create 1 JMS connections from the same connection factory.
              // Wait a little while to make sure broadcasts from all nodes have reached the client
            // Thread.sleep(5000);
               connectionA = connectionFactory.createConnection();
               ((ActiveMQConnection) connectionA).setFailoverListener(new FailoverListenerImpl());





               // Step 5. We create JMS Sessions
               Session sessionA = connectionA.createSession(false, Session.AUTO_ACKNOWLEDGE);

               Queue queue = (Queue) initialContext.lookup("jms/queue/ali");
               // Step 6. We create JMS MessageProducer objects on the sessions
               MessageProducer producerA = sessionA.createProducer(queue);

               // Step 7. We send some messages on each producer
               final int numMessages = 10000000;

              for (int i = 0; i < numMessages; i++) {

              TextMessage messageA = sessionA.createTextMessage("A:This is text message " + i);
               producerA.send(messageA);
               System.out.println("Sent message: " + messageA.getText());

               }

             

               // Step 8. We start the connection to consume messages
               connectionA.start();


               } finally {

               // Step 10. Be sure to close our resources!

               if (connectionA != null) {

              connectionA.close();
               }

             

               if (initialContext != null) {

              initialContext.close();
               }

             

              }

              • Actions
            • 3. Re: Why HornetQConnectionFactory client does not automatic failover?
              jbertram
              jbertram Jun 15, 2017 11:57 AM (in response to alireza.alallah)

              Based on the log information from the slave you provided I don't believe the backup broker actually activated when you shut down the master.  Are you sure the cluster is forming properly?

                • Actions
              • 4. Re: Why HornetQConnectionFactory client does not automatic failover?
                alireza.alallah
                alireza.alallah Jun 15, 2017 3:54 PM (in response to jbertram)

                I have configured wildfly10 in domain mode base on Clustering and Domain Setup Walkthrough - WildFly 10 - Project Documentation Editor

                my configuration files has been attached

                Thanks for help me.

                  • Actions
                • 5. Re: Why HornetQConnectionFactory client does not automatic failover?
                  jbertram
                  jbertram Jun 16, 2017 4:11 PM (in response to alireza.alallah)

                  My recommendation would be to start with a simple configuration and work up from there.  Instead of starting with colocated brokers in a Wildfly domain just configure a 2 standalone Wildfly instances one with an ActiveMQ Artemis master and the other with an ActiveMQ Artemis slave.  Once you get that working for your use-case then move to standalone colocated and then finally to a Wildfly domain.

                   

                  Also, please don't hijack other people's threads (e.g. here) or post the same question multiple places.  It tends to be counter-productive.

                    • Actions