1 2 Previous Next 15 Replies Latest reply on Jun 19, 2017 3:33 AM by Miroslav Novak

    Wildfly 10 - ActiveMQ Artemis Failover with Standalone client

    Vincent Sourin Newbie

      Hello,

       

      I've got two wildly 10 instances (lastest master) running on two different severs with ActiveMQ messaging configured with colocated replication :

       

      <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
                      <server name="default">
                          <cluster password="${jboss.messaging.cluster.password:vs1979anpasswdcluster}"/>
                          <replication-colocated excluded-connectors="netty-connector" request-backup="true">
                              <master check-for-live-server="true" cluster-name="my-cluster"/>
                              <slave cluster-name="my-cluster"/>
                          </replication-colocated>
                          <security-setting name="#">
                              <role name="guest" delete-non-durable-queue="true" create-non-durable-queue="true" create-durable-queue="true" consume="true" send="true"/>
                          </security-setting>
                          <address-setting name="#" redistribution-delay="1000" message-counter-history-day-limit="10" page-size-bytes="2097152" max-size-bytes="10485760" expiry-address="jms.queue.ExpiryQueue" dead-letter-address="jms.queue.DLQ"/>
                          <http-connector name="http-connector" endpoint="http-acceptor" socket-binding="http"/>
                          <http-connector name="http-connector-throughput" endpoint="http-acceptor-throughput" socket-binding="http">
                              <param name="batch-delay" value="50"/>
                          </http-connector>
                          <remote-connector name="netty-connector" socket-binding="messaging"/>
                          <in-vm-connector name="in-vm" server-id="0"/>
                          <http-acceptor name="http-acceptor" http-listener="default"/>
                          <http-acceptor name="http-acceptor-throughput" http-listener="default">
                              <param name="batch-delay" value="50"/>
                              <param name="direct-deliver" value="false"/>
                          </http-acceptor>
                          <remote-acceptor name="netty-acceptor" socket-binding="messaging"/>
                          <in-vm-acceptor name="in-vm" server-id="0"/>
                          <broadcast-group name="bg-group1" connectors="http-connector" jgroups-channel="activemq-cluster"/>
                          <discovery-group name="dg-group1" jgroups-channel="activemq-cluster"/>
                          <cluster-connection name="my-cluster" discovery-group="dg-group1" connector-name="http-connector" address="jms"/>
                          <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
                          <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
                          <jms-queue name="PanoramaScanPresses" entries="java:jboss/exported/jms/queue/PanoramaScanPresses"/>
                          <jms-topic name="TopicPanoramaScanPresses" entries="jms/topic/TopicPanoramaScanPresses java:jboss/exported/jms/topic/TopicPanoramaScanPresses"/>
                          <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
                          <connection-factory name="RemoteConnectionFactory" failover-on-initial-connection="true" reconnect-attempts="-1" block-on-acknowledge="true" client-failure-check-period="10000" ha="true" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector"/>
                          <pooled-connection-factory name="activemq-ra" transaction="xa" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm"/>
                      </server>
                  </subsystem>
      
      

       

      and I've got a standalone JMS client connecting to wildlfy :

       

      [...]
                   Properties env = new Properties();
                  env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");
                  env.put(Context.PROVIDER_URL, System.getProperty(Context.PROVIDER_URL, "http-remoting://as1.XXXXX.corp:8080,http-remoting://as2.XXXXX.corp:8080"));
                  env.put(Context.SECURITY_PRINCIPAL, System.getProperty("username", "user"));
                  env.put(Context.SECURITY_CREDENTIALS, System.getProperty("password", "pass"));
                  Context jndiContext = new InitialContext(env);
                 
                  connectionFactory = (ConnectionFactory) jndiContext.lookup("java:jms/RemoteConnectionFactory");
                  System.out.println(connectionFactory.toString());
                  queue = (Destination) jndiContext.lookup("java:jms/topic/TopicPanoramaScanPresses");
                  jndiContext.close();
                 
                  jmsContext = connectionFactory.createContext("user", "pass", JMSContext.AUTO_ACKNOWLEDGE);
      [...]
      
      

       

      When I kill one of my wildfly instance, I can see in the logs that the backup server becomes live in the second server. But the client received this error :

       

      WARN: AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
      
      

       

      and doesn't failover to the second server. If I relaunched the killed server, the client reconnects to that server.

       

      What bothers me it's the connection factory received by my client does not mention the second server available and I think that's why the client does not failover :

       

      System.out.println(connectionFactory.toString());
      => ActiveMQConnectionFactory [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEnabled=true&port=8080&httpPpgradeEndpoint=http-acceptor&host=10-10-2-134], discoveryGroupConfiguration=null], clientID=null, consumerWindowSize = 1048576, dupsOKBatchSize=1048576, transactionBatchSize=1048576, readOnly=false]
      
      

       

      Does someone can tell me how to configure the connection factory correctly to allow failover of the client ?

       

      Thanks.

       

      Vincent.

        • 1. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
          Justin Bertram Master

          When you say you "kill" the server what exactly are you doing? 

          • 2. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
            Vincent Sourin Newbie

            Hello Justin,

             

            I tried to 'kill -9 <PID>' and also to gracefully shutdown the instance of Wildfly where the client is connected.

            With both commands, I can see in wildfly logs that backup server become live but the client only received this error :

             

            WARN: AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
            

             

            and wait the killed / shutted down server to become live again to reconnect.

             

            Can you confirm that the connection factory looked up by JNDI is correct :

            ActiveMQConnectionFactory [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEnabled=true&port=8080&httpPpgradeEndpoint=http-acceptor&host=10-10-2-134], discoveryGroupConfiguration=null], clientID=null, consumerWindowSize = 1048576, dupsOKBatchSize=1048576, transactionBatchSize=1048576, readOnly=false]
            


            Are all servers should not be mentioned in connection ?

            • 3. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
              Justin Bertram Master

              Your configuration looks good as well as the output from ActiveMQConnectionFactory.toString().  The thing about ActiveMQConnectionFactory is that it's basically just a stub downloaded from JNDI on whatever server the lookup takes place.  Here's how the connection factory is configured on the server:

               

                <connection-factory name="RemoteConnectionFactory" failover-on-initial-connection="true" reconnect-attempts="-1" block-on-acknowledge="true" client-failure-check-period="10000" ha="true" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector"/>

               

              The "connectors" in the XML get translated into the "initialConnectors" you see in the ActiveMQConnectionFactory.  In your case, they both reflect a single connector.  This is what is used when a client invokes createConnection(..) from this instance.  However, once a connection is created then that connection will receive an update from the cluster informing it up the backup server and that information will be used for fail-over if the connection dies.  This is done automatically.  If the client closes the connection and goes back to use the original ActiveMQConnectionFactory then it will be forced to use the initialConnectors again which may be pointing to a dead server at that point.

               

              How exactly are you conducting your test?

              • 4. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
                Vincent Sourin Newbie

                Hello Justin,

                 

                Thanks for the explanations.

                 

                Here is how I conduct my test :

                 

                * I run my standalone test program and I produce two messages (which are consumed correctly by the consumer).

                * Then I killed (kill -9 <PID>) the wildfly instance where the test program is connected to. In the second wildfly instance, I can see that the backup server become live :

                [Server:as2] 09:33:32,176 INFO  [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null) AMQ221007: Server is now live
                [Server:as2] 09:33:32,206 INFO  [org.apache.activemq.artemis.core.server] (Thread-9 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$2@71f9c4b9-2134779126)) AMQ221027: Bridge ClusterConnectionBridge@477d6f72 [name=sf.my-cluster.d8870599-
                e3a3-11e5-b881-192c13464ad1, queue=QueueImpl[name=sf.my-cluster.d8870599-e3a3-11e5-b881-192c13464ad1, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=f65789c3-e3d1-11e5-9ab1-e3176aea142f]]@49ba99ed targetConnector=ServerLocatorImpl (identity=(Cluster-
                connection-bridge::ClusterConnectionBridge@477d6f72 [name=sf.my-cluster.d8870599-e3a3-11e5-b881-192c13464ad1, queue=QueueImpl[name=sf.my-cluster.d8870599-e3a3-11e5-b881-192c13464ad1, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=f65789c3-
                e3d1-11e5-9ab1-e3176aea142f]]@49ba99ed targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEnabled=true&httpPpgradeEndpoint=http-
                acceptor&port=8080&host=10-10-2-130], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1222539910[nodeUUID=f65789c3-e3d1-11e5-9ab1-e3176aea142f, connector=TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-
                NettyConnectorFactory) ?httpUpgradeEnabled=true&httpPpgradeEndpoint=http-acceptor&port=8180&host=10-10-2-130, address=jms, server=ActiveMQServerImpl::serverUUID=f65789c3-e3d1-11e5-9ab1-e3176aea142f])) [initialConnectors=[TransportConfiguration(name=http-connector,
                factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEnabled=true&httpPpgradeEndpoint=http-acceptor&port=8080&host=10-10-2-130], discoveryGroupConfiguration=null]] is connected
                
                

                 

                * At this moment, the test program received :

                WARN: AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
                
                

                and don't failover to the second server .

                * I restart the first instance of wildfly and the test program reconnects to this instance. I can produce messages again.

                 

                I launched the test program with DEBUG enabled and it seems that it receives the cluster topology correctly. But when the server is killed, the client complains that the backup server is not active whereas I see it becomes live in wildfly log.

                Here are the logs of the client :

                 

                09:56:20,322 DEBUG [org.jboss.logging] Logging Provider: org.jboss.logging.JBossLogManagerProvider
                09:56:20,550 DEBUG [org.jboss.naming.remote.client.InitialContextFactory] Looking for jboss-naming-client.properties using classloader sun.misc.Launcher$AppClassLoader@73d16e93
                09:56:20,559 DEBUG [org.jboss.naming.remote.client.InitialContextFactory] jboss.naming.client.endpoint.create.options. has the following options {}
                09:56:20,571 DEBUG [org.jboss.naming.remote.client.InitialContextFactory] jboss.naming.client.remote.connectionprovider.create.options. has the following options {}
                09:56:20,769 INFO  [org.xnio] XNIO version 3.3.4.Final
                09:56:21,144 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.3.4.Final
                09:56:21,613 DEBUG [org.xnio.nio] Started channel thread 'Remoting "config-based-naming-client-endpoint" Accept', selector sun.nio.ch.WindowsSelectorImpl@19349db8
                09:56:21,613 DEBUG [org.xnio.nio] Started channel thread 'Remoting "config-based-naming-client-endpoint" I/O-1', selector sun.nio.ch.WindowsSelectorImpl@4f9b92b
                09:56:21,708 INFO  [org.jboss.remoting] JBoss Remoting version 4.0.18.Final
                09:56:21,834 DEBUG [org.jboss.naming.remote.client.InitialContextFactory] jboss.naming.client.connect.options. has the following options {}
                09:56:22,998 DEBUG [org.jboss.remoting.remote.client] Client authentication failed for mechanism JBOSS-LOCAL-USER: javax.security.sasl.SaslException: Failed to read server challenge [Caused by java.io.FileNotFoundException: \opt\wildfly\domain\tmp\auth\local3271632781459508514.challenge (Le chemin d’accès spécifié est introuvable)]
                09:56:24,849 DEBUG [org.apache.activemq.artemis.logs] using dummy address ffffff93:ffffff81:5c:ffffffb5:ffffffbe:-70
                ActiveMQConnectionFactory [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEnabled=true&port=8080&httpPpgradeEndpoint=http-acceptor&host=10-10-2-134], discoveryGroupConfiguration=null], clientID=null, consumerWindowSize = 1048576, dupsOKBatchSize=1048576, transactionBatchSize=1048576, readOnly=false]
                09:56:25,095 DEBUG [org.jboss.naming.remote.protocol.v1.RemoteNamingStoreV1] Channel end notification received, closing channel Channel ID b5a07d45 (outbound) of Remoting connection 53d631d6 to as1.XXXXXXX.corp/10.10.2.134:8080
                09:56:25,212 DEBUG [org.apache.activemq.artemis.core.client] Trying reconnection attempt 0/1
                09:56:25,213 DEBUG [org.apache.activemq.artemis.core.client] Trying to connect with connector = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@eb21112, parameters = {httpUpgradeEnabled=true, port=8080, httpPpgradeEndpoint=http-acceptor, host=10.10.2.134} connector = null
                09:56:25,720 DEBUG [org.apache.activemq.artemis.core.client] Started Netty Connector version 4.0.32.Final
                09:56:25,721 DEBUG [org.apache.activemq.artemis.core.client] Remote destination: /10.10.2.134:8080
                09:56:26,241 DEBUG [org.apache.activemq.artemis.core.client] Sending HTTP request DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
                GET  HTTP/1.1
                Host: 10.10.2.134
                Upgrade: activemq-remoting
                Connection: Upgrade
                httpPpgradeEndpoint: http-acceptor
                Sec-ActiveMQRemoting-Key: TU5dEq3Tq1O9AVxkuHfFtA==
                09:56:26,628 DEBUG [org.apache.activemq.artemis.core.client] Reconnection successful
                09:56:26,654 DEBUG [org.apache.activemq.artemis.core.client] Setting up backup config = TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEnabled=true&httpPpgradeEndpoint=http-acceptor&port=8180&host=10-10-2-130 for live = TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEnabled=true&port=8080&httpPpgradeEndpoint=http-acceptor&host=10-10-2-134
                
                09:58:06,450 DEBUG [org.apache.activemq.artemis.core.client] Notifying f65789c3-e3d1-11e5-9ab1-e3176aea142f going down
                09:58:06,581 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
                09:58:06,639 DEBUG [org.apache.activemq.artemis.core.client] Trying reconnection attempt 0/-1
                09:58:06,641 DEBUG [org.apache.activemq.artemis.core.client] Trying to connect with connector = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@eb21112, parameters = {httpUpgradeEnabled=true, port=8080, httpPpgradeEndpoint=http-acceptor, host=10.10.2.134} connector = null
                09:58:06,647 DEBUG [org.apache.activemq.artemis.core.client] Started Netty Connector version 4.0.32.Final
                09:58:06,648 DEBUG [org.apache.activemq.artemis.core.client] Remote destination: /10.10.2.134:8080
                09:58:07,674 DEBUG [org.apache.activemq.artemis.core.client] Connector towards NettyConnector [host=10.10.2.134, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
                09:58:07,676 DEBUG [org.apache.activemq.artemis.core.client] Trying backup config = TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEnabled=true&httpPpgradeEndpoint=http-acceptor&port=8180&host=10-10-2-130
                09:58:07,678 DEBUG [org.apache.activemq.artemis.core.client] Started Netty Connector version 4.0.32.Final
                09:58:07,679 DEBUG [org.apache.activemq.artemis.core.client] Remote destination: /10.10.2.130:8180
                09:58:08,685 DEBUG [org.apache.activemq.artemis.core.client] Connector towards NettyConnector [host=10.10.2.130, port=8180, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
                09:58:08,687 DEBUG [org.apache.activemq.artemis.core.client] Started Netty Connector version 4.0.32.Final
                09:58:08,690 DEBUG [org.apache.activemq.artemis.core.client] Remote destination: /10.10.2.130:8180
                09:58:09,703 DEBUG [org.apache.activemq.artemis.core.client] Connector towards NettyConnector [host=10.10.2.130, port=8180, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
                09:58:09,706 DEBUG [org.apache.activemq.artemis.core.client] Backup is not active.
                09:58:09,707 WARN  [org.apache.activemq.artemis.core.client] AMQ212006: Waiting 2 000 milliseconds before next retry. RetryInterval=2 000 and multiplier=1
                09:58:11,711 DEBUG [org.apache.activemq.artemis.core.client] Trying reconnection attempt 1/-1
                09:58:11,712 DEBUG [org.apache.activemq.artemis.core.client] Trying to connect with connector = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@eb21112, parameters = {httpUpgradeEnabled=true, port=8080, httpPpgradeEndpoint=http-acceptor, host=10.10.2.134} connector = null
                09:58:11,714 DEBUG [org.apache.activemq.artemis.core.client] Started Netty Connector version 4.0.32.Final
                09:58:11,715 DEBUG [org.apache.activemq.artemis.core.client] Remote destination: /10.10.2.134:8080
                09:58:12,722 DEBUG [org.apache.activemq.artemis.core.client] Connector towards NettyConnector [host=10.10.2.134, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
                09:58:12,725 DEBUG [org.apache.activemq.artemis.core.client] Trying backup config = TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEnabled=true&httpPpgradeEndpoint=http-acceptor&port=8180&host=10-10-2-130
                09:58:12,727 DEBUG [org.apache.activemq.artemis.core.client] Started Netty Connector version 4.0.32.Final
                09:58:12,727 DEBUG [org.apache.activemq.artemis.core.client] Remote destination: /10.10.2.130:8180
                09:58:13,740 DEBUG [org.apache.activemq.artemis.core.client] Connector towards NettyConnector [host=10.10.2.130, port=8180, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
                09:58:13,742 DEBUG [org.apache.activemq.artemis.core.client] Started Netty Connector version 4.0.32.Final
                09:58:13,743 DEBUG [org.apache.activemq.artemis.core.client] Remote destination: /10.10.2.130:8180
                09:58:14,749 DEBUG [org.apache.activemq.artemis.core.client] Connector towards NettyConnector [host=10.10.2.130, port=8180, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
                09:58:14,752 DEBUG [org.apache.activemq.artemis.core.client] Backup is not active.
                09:58:14,753 WARN  [org.apache.activemq.artemis.core.client] AMQ212006: Waiting 2 000 milliseconds before next retry. RetryInterval=2 000 and multiplier=1
                09:58:16,756 DEBUG [org.apache.activemq.artemis.core.client] Trying reconnection attempt 2/-1
                09:58:16,766 DEBUG [org.apache.activemq.artemis.core.client] Trying to connect with connector = org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory@eb21112, parameters = {httpUpgradeEnabled=true, port=8080, httpPpgradeEndpoint=http-acceptor, host=10.10.2.134} connector = null
                09:58:16,768 DEBUG [org.apache.activemq.artemis.core.client] Started Netty Connector version 4.0.32.Final
                09:58:16,769 DEBUG [org.apache.activemq.artemis.core.client] Remote destination: /10.10.2.134:8080
                09:58:17,775 DEBUG [org.apache.activemq.artemis.core.client] Connector towards NettyConnector [host=10.10.2.134, port=8080, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
                09:58:17,777 DEBUG [org.apache.activemq.artemis.core.client] Trying backup config = TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEnabled=true&httpPpgradeEndpoint=http-acceptor&port=8180&host=10-10-2-130
                09:58:17,778 DEBUG [org.apache.activemq.artemis.core.client] Started Netty Connector version 4.0.32.Final
                09:58:17,779 DEBUG [org.apache.activemq.artemis.core.client] Remote destination: /10.10.2.130:8180
                09:58:18,784 DEBUG [org.apache.activemq.artemis.core.client] Connector towards NettyConnector [host=10.10.2.130, port=8180, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
                09:58:18,786 DEBUG [org.apache.activemq.artemis.core.client] Started Netty Connector version 4.0.32.Final
                09:58:18,787 DEBUG [org.apache.activemq.artemis.core.client] Remote destination: /10.10.2.130:8180
                09:58:19,785 DEBUG [org.apache.activemq.artemis.core.client] Connector towards NettyConnector [host=10.10.2.130, port=8180, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] failed
                09:58:19,787 DEBUG [org.apache.activemq.artemis.core.client] Backup is not active.
                09:58:19,789 WARN  [org.apache.activemq.artemis.core.client] AMQ212006: Waiting 2 000 milliseconds before next retry. RetryInterval=2 000 and multiplier=1
                [...]
                
                
                • 5. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
                  Miroslav Novak Master

                  Hi Vincent,

                   

                  configuration of replicated-colocated topology with http connectors and acceptors is not well supported. The problem here might be that live and backup (this is backup requested by the other live in cluster) are both connected to the same http listener defined in undertow and have the same http upgrade header.

                   

                  I would suggest to configure colocated backup in the old way like using remote connectors and acceptors like:

                      <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
                          <server name="default">
                              <security enabled="false"/>
                              <cluster password="${jboss.messaging.cluster.password:CHANGE ME!!}"/>
                              <journal compact-min-files="0" min-files="10"/>
                              <replication-master check-for-live-server="true" group-name="group1" cluster-name="my-cluster"/>
                              <bindings-directory path="../../../../hornetq-journal-A/bindings"/>
                              <journal-directory path="../../../../hornetq-journal-A/journal"/>
                              <large-messages-directory path="../../../../hornetq-journal-A/largemessages"/>
                              <paging-directory path="../../../../hornetq-journal-A/paging"/>
                              <security-setting name="#">
                                  <role name="guest" delete-non-durable-queue="true" create-non-durable-queue="true" consume="true" send="true"/>
                              </security-setting>
                              <address-setting name="#" redistribution-delay="0" page-size-bytes="524288" max-size-bytes="1048576" max-delivery-attempts="200"/>
                              <http-connector name="http-connector" endpoint="http-acceptor" socket-binding="http"/>
                              <http-connector name="http-connector-throughput" endpoint="http-acceptor-throughput" socket-binding="http">
                                  <param name="batch-delay" value="50"/>
                              </http-connector>
                              <in-vm-connector name="in-vm" server-id="0"/>
                              <http-acceptor name="http-acceptor" http-listener="default"/>
                              <http-acceptor name="http-acceptor-throughput" http-listener="default">
                                  <param name="batch-delay" value="50"/>
                                  <param name="direct-deliver" value="false"/>
                              </http-acceptor>
                              <in-vm-acceptor name="in-vm" server-id="0"/>
                              <broadcast-group name="bg-group1" connectors="http-connector" jgroups-channel="activemq-cluster"/>
                              <discovery-group name="dg-group1" refresh-timeout="1000" jgroups-channel="activemq-cluster" jgroups-stack="udp"/>
                              <cluster-connection name="my-cluster" discovery-group="dg-group1" connector-name="http-connector" address="jms"/>
                              <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
                              <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
                              <jms-queue name="testQueue0" entries="jms/queue/testQueue0 java:jboss/exported/jms/queue/testQueue0"/>
                              <jms-queue name="InQueue" entries="jms/queue/InQueue java:jboss/exported/jms/queue/InQueue"/>
                              <jms-queue name="OutQueue" entries="jms/queue/OutQueue java:jboss/exported/jms/queue/OutQueue"/>
                              <jms-topic name="testTopic0" entries="jms/topic/testTopic0 java:jboss/exported/jms/topic/testTopic0"/>
                              <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
                              <connection-factory name="RemoteConnectionFactory" reconnect-attempts="-1" retry-interval-multiplier="1.0" retry-interval="1000" block-on-acknowledge="true" ha="true" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector"/>
                              <pooled-connection-factory name="activemq-ra" transaction="xa" reconnect-attempts="-1" block-on-acknowledge="true" ha="true" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm"/>
                          </server>
                          <server name="backup">
                              <security enabled="false"/>
                              <cluster password="CHANGE_ME!!!"/>
                              <replication-slave max-saved-replicated-journal-size="-1" group-name="group2" cluster-name="my-cluster">
                                  <scale-down enabled="false"/>
                              </replication-slave>
                              <bindings-directory path="../../../../hornetq-journal-B/bindings"/>
                              <journal-directory path="../../../../hornetq-journal-B/journal"/>
                              <large-messages-directory path="../../../../hornetq-journal-B/largemessages"/>
                              <paging-directory path="../../../../hornetq-journal-B/paging"/>
                              <address-setting name="#" redistribution-delay="0" page-size-bytes="524288" max-size-bytes="1048576" max-delivery-attempts="200"/>
                              <remote-connector name="netty-backup" socket-binding="messaging-backup"/>
                              <in-vm-connector name="in-vm" server-id="0"/>
                              <remote-acceptor name="netty-backup" socket-binding="messaging-backup"/>
                              <broadcast-group name="bg-group-backup" connectors="netty-backup" broadcast-period="1000" jgroups-channel="activemq-cluster" jgroups-stack="udp"/>
                              <discovery-group name="dg-group-backup" refresh-timeout="1000" jgroups-channel="activemq-cluster" jgroups-stack="udp"/>
                              <cluster-connection name="my-cluster" discovery-group="dg-group-backup" retry-interval="1000" connector-name="netty-backup" address="jms"/>
                          </server>
                      </subsystem>
                  • 6. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
                    Vincent Sourin Newbie

                    Hello Miroslav,

                     

                    Using the "old way" works like a charm. My client now failovers correctly.

                     

                    A big thank to you and Justin for your help and your time.

                    • 7. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
                      Miroslav Novak Master

                      You're welcome! Happy that it works for you :-)

                      • 8. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
                        Benjamin Buehlmann Newbie

                        Hi Miroslav,

                        Should the failover work with http connectors in a shared-store-master/-slave setup? On the client-side I've defined two connectors on the pooled-connection-factory. One connector to the master broker and the other connector to the slave broker. When I kill the active broker, no failover happens..

                         

                        <pooled-connection-factory name="remote-artemis" transaction="xa" ha="true" entries="java:/JmsXA java:/jms/RemoteArtemisConnectionFactory java:jboss/DefaultJMSConnectionFactory" block-on-acknowledge="true" connectors="remote-http-connector-active remote-http-connector-backup" user="user" password="pass" client-failure-check-period="10000" reconnect-attempts="-1" retry-interval="1000"/>
                        • 9. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
                          Miroslav Novak Master

                          Hi Benjamin,

                           

                          sorry for late reply, I had vacation. Yes, it should work. Could you share your xml config?

                           

                          Thanks,

                          Mirek

                          • 10. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
                            Benjamin Buehlmann Newbie

                            Hi Mirek,

                            Finally it worked. I was not aware that it is mandatory to configure the Master-Slave Pair as a Cluster.

                            Cheers,

                            Benjamin

                            • 11. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
                              Miroslav Novak Master

                              Great to hear that you made it work.

                               

                              Yes, all servers (live-backup pairs) must be in the same artemis cluster so they know the whole topology. Any client which connects to this cluster then receives topology update so in case that live server crashes then it knows where its backup is. Then client will failover to this backup.

                               

                              You can remove connector "remote-http-connector-backup" from configuration of pooled-connection-factory. Client (in this case MDB or EJB which is using this pooled-connection-factory) will automatically get topology update after first successful connection.

                               

                              Thanks,

                              Mirek

                              • 12. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
                                alireza alallah Newbie

                                Hi

                                I have same problem but I can't solve problem with your solution , my client thrown exception as below

                                 

                                Jun 15, 2017 6:42:05 PM org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl fail

                                WARN: AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]

                                Failover event triggered :FAILURE_DETECTED

                                Jun 15, 2017 6:42:09 PM org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl getConnectionWithRetry

                                WARN: AMQ212005: Tried 5 times to connect. Now giving up on reconnecting it.

                                Failover event triggered :FAILOVER_FAILED

                                Exception in thread "main" javax.jms.JMSException: AMQ119016: Connection failure detected. Unblocking a blocking call that will never get a response

                                  at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:406)

                                  at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:304)

                                  at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:378)

                                  at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:286)

                                  at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:264)

                                  at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:122)

                                  at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:477)

                                  at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:173)

                                  at Application.main(Application.java:65)

                                Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED message=AMQ119016: Connection failure detected. Unblocking a blocking call that will never get a response]

                                  ... 9 more

                                 

                                my config as below:

                                 

                                master config:

                                <server name="default">

                                                    <security enabled="false"/>

                                                    <cluster password="${jboss.messaging.cluster.password:sh@1396SM}"/>

                                 

                                 

                                                    <replication-master check-for-live-server="true" group-name="ali-server-group" cluster-name="my-cluster"/>

                                                    <security-setting name="#">

                                                        <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>

                                                    </security-setting>

                                                    <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>

                                                    <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>

                                                    <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">

                                                        <param name="batch-delay" value="50"/>

                                                    </http-connector>

                                                    <in-vm-connector name="in-vm" server-id="0"/>

                                                    <http-acceptor name="http-acceptor" http-listener="default"/>

                                                    <http-acceptor name="http-acceptor-throughput" http-listener="default">

                                                        <param name="batch-delay" value="50"/>

                                                        <param name="direct-deliver" value="false"/>

                                                    </http-acceptor>

                                                    <in-vm-acceptor name="in-vm" server-id="0"/>

                                                    <broadcast-group name="bg-group1" jgroups-channel="activemq-cluster" connectors="http-connector"/>

                                                    <discovery-group name="dg-group1" jgroups-channel="activemq-cluster"/>

                                                    <cluster-connection name="my-cluster" address="jms" connector-name="http-connector" discovery-group="dg-group1"/>

                                                    <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>

                                                    <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>

                                                    <jms-queue name="ali" entries="java:jboss/exported/jms/queue/ali"/>

                                                    <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>

                                                    <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" retry-interval-multiplier="1.0" retry-interval="1000" client-failure-check-period="1000" failover-on-initial-connection="true"  discovery-group="dg-group1" ha="true" block-on-acknowledge="true" reconnect-attempts="5"/>

                                                    <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" transaction="xa" reconnect-attempts="-1" block-on-acknowledge="true" ha="true" />

                                                </server>

                                                <server name="backup">

                                                <security enabled="false"/>

                                                <cluster password="sh@1396SM"/>

                                                <replication-slave max-saved-replicated-journal-size="-1" group-name="group2" cluster-name="my-cluster">

                                                <scale-down enabled="false"/>

                                                </replication-slave>

                                                <address-setting name="#" redistribution-delay="0" page-size-bytes="524288" max-size-bytes="1048576" max-delivery-attempts="200"/>

                                                <remote-connector name="netty-backup" socket-binding="messaging-backup"/>

                                                <in-vm-connector name="in-vm" server-id="0"/>

                                                <remote-acceptor name="netty-backup" socket-binding="messaging-backup"/>

                                                <broadcast-group name="bg-group-backup" connectors="netty-backup" broadcast-period="1000" jgroups-channel="activemq-cluster" jgroups-stack="udp"/>

                                                <discovery-group name="dg-group-backup" refresh-timeout="1000" jgroups-channel="activemq-cluster" jgroups-stack="udp"/>

                                                <cluster-connection name="my-cluster" discovery-group="dg-group-backup" retry-interval="1000" connector-name="netty-backup" address="jms"/>

                                                </server>

                                 

                                my slave config as below:

                                <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">

                                                <server name="default">

                                                    <security enabled="false"/>

                                                    <cluster password="${jboss.messaging.cluster.password:sh@1396SM}"/>

                                                    <security-setting name="#">

                                                        <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>

                                                    </security-setting>

                                                    <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>

                                                    <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>

                                                    <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">

                                                        <param name="batch-delay" value="50"/>

                                                    </http-connector>

                                                    </connector>

                                                    <replication-master check-for-live-server="true" group-name="ali-server-group" cluster-name="my-cluster"/>

                                                    <in-vm-connector name="in-vm" server-id="0"/>

                                                    <http-acceptor name="http-acceptor" http-listener="default"/>

                                                    <http-acceptor name="http-acceptor-throughput" http-listener="default">

                                                        <param name="batch-delay" value="50"/>

                                                        <param name="direct-deliver" value="false"/>

                                                    </http-acceptor>

                                                    <remote-connector name="netty-backup" socket-binding="messaging-backup"/>

                                                    <remote-acceptor name="netty-backup" socket-binding="messaging-backup"/>

                                                    <in-vm-acceptor name="in-vm" server-id="0"/>

                                                    <broadcast-group name="bg-group1" jgroups-channel="activemq-cluster" connectors="http-connector"/>

                                                    <discovery-group name="dg-group1" jgroups-channel="activemq-cluster"/>

                                                    <cluster-connection name="my-cluster" address="jms" connector-name="http-connector" discovery-group="dg-group1" failover-on-initial-connection="true"/>

                                                    <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>

                                                    <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>

                                                    <jms-queue name="ali" entries="java:jboss/exported/jms/queue/ali"/>

                                                    <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>

                                                    <connection-factory name="RemoteConnectionFactory"  entries="java:jboss/exported/jms/RemoteConnectionFactory" failover-on-initial-connection="true" client-failure-check-period="1000" discovery-group="dg-group1"  ha="true" retry-interval-multiplier="1.0" retry-interval="1000" block-on-acknowledge="true" reconnect-attempts="5"/>

                                                    <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" reconnect-attempts="-1" block-on-acknowledge="true" ha="true" connectors="in-vm" transaction="xa" block-on-acknowledge="true" />

                                                </server>

                                                <server name="backup">

                                                <security enabled="false"/>

                                                <cluster password="sh@1396SM"/>

                                                <replication-slave max-saved-replicated-journal-size="-1" group-name="group2" cluster-name="my-cluster">

                                                <scale-down enabled="false"/>

                                                </replication-slave>

                                                <address-setting name="#" redistribution-delay="0" page-size-bytes="524288" max-size-bytes="1048576" max-delivery-attempts="200"/>

                                                <remote-connector name="netty-backup" socket-binding="messaging-backup"/>

                                                <in-vm-connector name="in-vm" server-id="0"/>

                                                <remote-acceptor name="netty-backup" socket-binding="messaging-backup"/>

                                                <broadcast-group name="bg-group-backup" connectors="netty-backup" broadcast-period="1000" jgroups-channel="activemq-cluster" jgroups-stack="udp"/>

                                                <discovery-group name="dg-group-backup" refresh-timeout="1000" jgroups-channel="activemq-cluster" jgroups-stack="udp"/>

                                                <cluster-connection name="my-cluster" discovery-group="dg-group-backup" retry-interval="1000" connector-name="netty-backup" address="jms"/>

                                                </server>

                                            </subsystem>

                                 

                                 

                                my client code as below:

                                 

                                 

                                InitialContext initialContext = null;

                                   Connection connectionA = null;

                                  try {

                                   final Properties props = new Properties();
                                   props.put("java.naming.factory.initial",
                                   "org.jboss.naming.remote.client.InitialContextFactory");
                                   props.put("java.naming.provider.url",
                                   "http-remoting://192.168.0.200:8330,http-remoting://192.168.0.201:8330");


                                   props.put(Context.SECURITY_PRINCIPAL, "isc");
                                   // password
                                   props.put(Context.SECURITY_CREDENTIALS, "sh@1396SM");

                                   // Step 1. Get an initial context for looking up JNDI from server 0
                                   initialContext = new InitialContext(props);

                                   // Step 2. Look-up the JMS Queue object from JNDI

                                  // Step 3. Look-up a JMS Connection Factory object from JNDI on server 0
                                   ConnectionFactory connectionFactory = (ConnectionFactory) initialContext.lookup("jms/RemoteConnectionFactory");



                                   ((ActiveMQConnectionFactory)connectionFactory).setInitialConnectAttempts(20);
                                   ((ActiveMQConnectionFactory)connectionFactory).setFailoverOnInitialConnection(true);
                                   // Step 4. We create 1 JMS connections from the same connection factory.
                                  // Wait a little while to make sure broadcasts from all nodes have reached the client
                                // Thread.sleep(5000);
                                   connectionA = connectionFactory.createConnection();
                                   ((ActiveMQConnection) connectionA).setFailoverListener(new FailoverListenerImpl());





                                   // Step 5. We create JMS Sessions
                                   Session sessionA = connectionA.createSession(false, Session.AUTO_ACKNOWLEDGE);

                                   Queue queue = (Queue) initialContext.lookup("jms/queue/ali");
                                   // Step 6. We create JMS MessageProducer objects on the sessions
                                   MessageProducer producerA = sessionA.createProducer(queue);

                                   // Step 7. We send some messages on each producer
                                   final int numMessages = 10000000;

                                  for (int i = 0; i < numMessages; i++) {

                                  TextMessage messageA = sessionA.createTextMessage("A:This is text message " + i);
                                   producerA.send(messageA);
                                   System.out.println("Sent message: " + messageA.getText());

                                   }

                                 

                                   // Step 8. We start the connection to consume messages
                                   connectionA.start();


                                   } finally {

                                   // Step 10. Be sure to close our resources!

                                   if (connectionA != null) {

                                  connectionA.close();
                                   }

                                 

                                   if (initialContext != null) {

                                  initialContext.close();
                                   }

                                 

                                  }

                                • 13. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
                                  Miroslav Novak Master

                                  Hi Alireza,

                                   

                                  configuration was for one server. Config for 2nd server has little change. As it's required to tell Artemis master with which backup it will pair, it's necessary to set the same group-name="firstPair" in configuration of master and slave. In your configuration I can see that you set the same for both of the masters and then slaves so it's basically pairing master with master and slave with slave which is not correct.

                                   

                                  Also set journal path for backup server to different location by:

                                                <bindings-directory path="../../../../hornetq-journal-B/bindings"/>

                                              <journal-directory path="../../../../hornetq-journal-B/journal"/>
                                              <large-messages-directory path="../../../../hornetq-journal-B/largemessages"/>
                                              <paging-directory path="../../../../hornetq-journal-B/paging"/>

                                  Otherwise master and backup in one WF10 server will use the same journal directory which would be really bad :-) I'm not sure if your client is able to survive failover as I've neve tried to use failover listener. Basically when JMSException is thrown, you client should catch it and retry last operation. In this Artemis client library does not shield client from handling all types of failures which can happen. It just retries connection to master-slave pair. If any of them is up then it connects to it. Also once this happens then your current session will not remember any message send/receive before failover/failiback. This is the reason why last operation should be retried.

                                   

                                  In short what client should handle is:

                                  - If it's producer and is sending messages then when JMSException/TransactionRollbackException is thrown then is should try to resend messages. Set _AMQ_DUP_ID to the messages so server can filter duplicates.

                                  - If it's consumer then it's more complicated to guarantee once and only once delivery with simple JMS API. Best is to use MDB consuming messages in XA transaction as it will deal with all types of failures.

                                  -- With simple JMS consumer It's possible if you have just one consumer on queue and with TRANSACTED_SESSION. If consumer gets JMSException for session.commit() call then it's not clear, whether messages were commited or not. In this case you need to hold _AMQ_DUP_IDs of messages in client cache and check that if you get duplicates then you don't process messages on client side. If TransactionRollbackException is thrown than just repeat comsumption of messages.

                                   

                                  Thanks,

                                  Mirek

                                  1 of 1 people found this helpful
                                  • 14. Re: Wildfly 10 - ActiveMQ Artemis Failover with Standalone client
                                    alireza alallah Newbie

                                    Hi Miroslav,

                                    Thanks for reply and good explanation, I have applied your new configuration , and for one sever (colocated server) works , but i want to have automatic failover between two seprate server my client can't reconnet to server2 and thrown exception as below:

                                     

                                    Jun 16, 2017 6:05:36 PM org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl getConnectionWithRetry

                                    WARN: AMQ212005: Tried 5 times to connect. Now giving up on reconnecting it.

                                    Exception in thread "main" javax.jms.JMSException: AMQ119016: Connection failure detected. Unblocking a blocking call that will never get a response

                                      at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:406)

                                      at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:304)

                                      at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:378)

                                      at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:286)

                                      at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:264)

                                      at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:122)

                                      at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:477)

                                      at org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:173)

                                      at Application.main(Application.java:66)

                                    Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED message=AMQ119016: Connection failure detected. Unblocking a blocking call that will never get a response]

                                      ... 9 more

                                     

                                    and server two log as below:

                                     

                                    [Host Controller] 09:50:18,769 WARN  [org.jboss.as.host.controller] (management task-4) WFLYHC0015: Connection to remote host-controller closed. Trying to reconnect.

                                    [Host Controller] 09:50:18,770 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 14) WFLYHC0150: Trying to reconnect to master host controller.

                                    [Server:server-three] 09:50:18,901 DEBUG [org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl] (Thread-1 (ActiveMQ-client-netty-threads-1729988865)) Invocation of interceptor org.apache.activemq.artemis.core.server.impl.ReplicationError on PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=60000] returned true

                                    [Server:server-three] 09:50:18,940 DEBUG [org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl] (Thread-2 (ActiveMQ-client-netty-threads-1729988865)) Invocation of interceptor org.apache.activemq.artemis.core.server.impl.ReplicationError on PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=60000] returned true

                                    [Host Controller] 09:50:19,788 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 14) WFLYHC0150: Trying to reconnect to master host controller.

                                    [Server:server-three] 09:50:22,654 DEBUG [org.jboss.activemq.artemis.wildfly.integration.recovery] (Periodic Recovery)

                                    [Server:server-three] =======================================================================================

                                    [Server:server-three] 09:50:22,655 DEBUG [org.jboss.activemq.artemis.wildfly.integration.recovery] (Periodic Recovery) Returning the following list on getXAREsources:

                                    [Server:server-three] 09:50:22,655 DEBUG [org.jboss.activemq.artemis.wildfly.integration.recovery] (Periodic Recovery) server-id=d950343e-5276-11e7-baf3-c9eb41dd3164, value=org.jboss.activemq.artemis.wildfly.integration.WildFlyActiveMQXAResourceWrapper@51691a9

                                    [Server:server-three] 09:50:22,655 DEBUG [org.jboss.activemq.artemis.wildfly.integration.recovery] (Periodic Recovery) =======================================================================================

                                    [Server:server-three]

                                    [Server:server-three] 09:50:22,655 DEBUG [org.jboss.ejb.client.txn] (Periodic Recovery) Send recover request for transaction origin node identifier 1 to EJB receiver with node name slave:server-three

                                    [Server:server-three] 09:50:22,655 DEBUG [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) looking for recover at ClientSessionImpl [name=b73d0ba8-5297-11e7-8153-e9373f923ef0, username=null, closed=false, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@6fdcac65, metaData=()]@4731dc24 configuration [XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA]]

                                    [Host Controller] 09:50:23,802 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 14) WFLYHC0150: Trying to reconnect to master host controller.

                                    [Server:server-three] 09:50:32,167 WARN  [org.jgroups.protocols.UDP] (thread-1,activemq-cluster,slave:server-three) JGRP000012: discarded message from different cluster ee (our cluster is activemq-cluster). Sender was slave:server-three (received 15 identical messages from slave:server-three in the last 63965 ms)

                                    [Server:server-three] 09:50:32,657 DEBUG [org.jboss.ejb.client.txn] (Periodic Recovery) Send recover request for transaction origin node identifier 1 to EJB receiver with node name slave:server-three

                                    [Server:server-three] 09:50:32,657 DEBUG [org.apache.activemq.artemis.service.extensions.xa.recovery] (Periodic Recovery) looking for recover at ClientSessionImpl [name=b73d0ba8-5297-11e7-8153-e9373f923ef0, username=null, closed=false, factory = org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl@6fdcac65, metaData=()]@4731dc24 configuration [XARecoveryConfig [transportConfiguration=[TransportConfiguration(name=, factory=org-apache-activemq-artemis-core-remoting-impl-invm-InVMConnectorFactory) ?serverId=0], discoveryConfiguration=null, username=null, password=****, JNDI_NAME=java:/JmsXA]]

                                    [Host Controller] 09:50:32,815 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 14) WFLYHC0150: Trying to reconnect to master host controller.

                                    [Server:server-three] 09:50:35,328 WARN  [org.jgroups.protocols.UDP] (thread-1,ee,slave:server-three) JGRP000012: discarded message from different cluster activemq-cluster (our cluster is ee). Sender was slave:server-three (received 72 identical messages from slave:server-three in the last 60023 ms)

                                    1 2 Previous Next