Symmetric cluster replication over network

[raivil]

Hi,

I'm trying to deploy a symmetric cluster using two physical servers.

Using two hornetq instances on the same machine works perfectly. The data is replicated on the nodes. If one node dies the application can continue to work using other instance. I can add a new instances to the cluster. Perfect.

But, using a remote instance doesnt seems to work. The bridges are created, but the data is not replicated, If the node that the application is connected dies, the app cant find the remote node to continue.

Is there anything wrong with the configuration? Did i miss something?

Client config.

<bean id="jmsConnectionFactoryUserCredential" primary="true" class="org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter">
   <property name="targetConnectionFactory" ref="hornetConnectionFactory"/>
   <property name="username" value="username"/>
   <property name="password" value="password"/>
  </bean>

<bean id="hornetConnectionFactory" class="org.hornetq.api.jms.HornetQJMSClient" factory-method="createConnectionFactoryWithHA">
        <constructor-arg index="0">
            <util:constant id="CF"  static-field="org.hornetq.api.jms.JMSFactoryType.CF" /> 
        </constructor-arg>
        <constructor-arg index="1">
            <bean class="org.hornetq.api.core.TransportConfiguration">
                <constructor-arg value="org.hornetq.core.remoting.impl.netty.NettyConnectorFactory" />
                <constructor-arg>
                    <map key-type="java.lang.String" value-type="java.lang.Object">    
                        <entry key="host" value="127.0.0.1"></entry>
                        <entry key="port" value="5445"></entry>
                    </map>
                </constructor-arg>
            </bean>
        </constructor-arg>
    </bean>

Server configuration.

<connectors>
        <connector name="inVM">
            <factory-class>org.hornetq.core.remoting.impl.invm.InVMConnectorFactory</factory-class>
        </connector>


        <connector name="netty">
            <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
            <param key="port" value="${hornetq.remoting.netty.port:5445}" />
        </connector>


    </connectors>


    <acceptors>
        <acceptor name="inVM">
            <factory-class>org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory</factory-class>
        </acceptor>


        <acceptor name="netty">
            <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
            <param key="port" value="${hornetq.remoting.netty.port:5445}" />
        </acceptor>


    </acceptors>


  <!-- Clustering configuration -->
   <broadcast-groups>
      <broadcast-group name="my-broadcast-group">
         <group-address>${udp-address:224.0.0.1}</group-address>
         <group-port>9876</group-port>
         <broadcast-period>1000</broadcast-period>
         <connector-ref>netty</connector-ref>
      </broadcast-group>
   </broadcast-groups>
   
   <discovery-groups>
      <discovery-group name="my-discovery-group">
         <group-address>${udp-address:224.0.0.1}</group-address>
         <group-port>9876</group-port>
         <refresh-timeout>10000</refresh-timeout>
      </discovery-group>
   </discovery-groups>
   
   <cluster-connections>
      <cluster-connection name="my-cluster">
         <address>jms</address>
         <connector-ref>netty</connector-ref>
         <retry-interval>500</retry-interval>
         <use-duplicate-detection>true</use-duplicate-detection>
         <forward-when-no-consumers>true</forward-when-no-consumers>
         <max-hops>1</max-hops>
         <discovery-group-ref discovery-group-name="my-discovery-group"/>
  </cluster-connection>
   </cluster-connections>

[jbertram]

Can you clarify your configuration a bit?  You say that you're using a "symmetric cluster" and you provide some details about the server configuration.  However, you also say that you expect replication between the servers even though the server configuration doesn't contain the parameters to enable such functionality.  Replication is only supported between a live and a backup which, by definition, have different configurations (i.e. non-symmetric).  It's possible to co-locate a live and a backup in the same JVM (e.g. in Wildfly) so that you can achieve a truly symmetric configuration with replication, but I don't see any evidence for that in your configuration.

[raivil]

The configuration is basically copied from the example "symmetric-cluster".

From my understanding, a message sent to one node should be available on the other nodes os the cluster. That's what I called replication.

The replication between live and backup servers is not important at this time.

The problem i'm having is using the  symmetric-cluster example over the network. The example uses instances on the same machine.

I'm trying to use a remote node, which join the cluster but doenst seems to be participating.

[jbertram]

From my understanding, a message sent to one node should be available on the other nodes os the cluster. That's what I called replication.

The term replication implies a copy is being made (i.e. a replica).  This is not what happens in the scenario you describe, and therefore should not be referred to as replication.  What you describe is called message redistribution.

The problem i'm having is using the  symmetric-cluster example over the network. The example uses instances on the same machine.

I'm trying to use a remote node, which join the cluster but doenst seems to be participating.

In what way specifically does it seem not to be participating?  I'm still unclear on that point.

Previously you said, "If the node that the application is connected dies, the app cant find the remote node to continue."  I would expect that behavior given your scenario since fail-over is only supported between a live and a backup (which you don't have).  In order to connect to the other cluster node your client would need to be written to handle that scenario (i.e. reconnect programmatically).  You can, of course, configure the client to try to reconnect to the node that went down so that when the node comes back up the client can continue executing normally.

[raivil]

Ok, let's talk about the distribution os messages.

The clients are configured using HornetQJMSClient.createConnectionFactoryWithHA, which javadoc states:

/**
    * Create a HornetQConnectionFactory which will receive cluster topology updates from the cluster
    * as servers leave or join and new backups are appointed or removed.
    * <p>
    * The initial list of servers supplied in this method is simply to make an initial connection to
    * the cluster, once that connection is made, up to date cluster topology information is
    * downloaded and automatically updated whenever the cluster topology changes. If the topology
    * includes backup servers that information is also propagated to the client so that it can know
    * which server to failover onto in case of live server failure.
    * @param initialServers The initial set of servers used to make a connection to the cluster.
    *           Each one is tried in turn until a successful connection is made. Once a connection
    *           is made, the cluster topology is downloaded and the rest of the list is ignored.
    * @return the HornetQConnectionFactory
 public static HornetQConnectionFactory createConnectionFactoryWithHA(...........

The client knows the cluster topology and if the topology changes, is abe to continue working, since it can reconnect to another instance/node.

I already have that working, but with all nodes on the same machine (symmetric-cluster example, basically, with diferent connectors port).

I tried to put one node over the network. it joins the cluster (Bridge Connection is created, JMX info shows the correct number of cluster nodes).

But when the topology changes, the client fails to reconnect to a remote node. Perhaps something missing in the configuration.

[jbertram]

I'll try to explain this again...

Automatic fail-over is only supported between a live and a backup.  You don't have a live and a backup.  You just have 2 live servers.  Therefore your client will not automatically failover.  Notice that the JavaDoc for HornetQJMSClient.createConnectionFactoryWithHA states, "If the topology includes backup servers that information is also propagated to the client so that it can know which server to failover onto in case of live server failure."  Your topology doesn't include a backup server so when a failure occurs the client can't failover automatically. 

In order for your client to reconnect to the other live node you'll need specifically code it for such a scenario.

Note, we are working on new functionality to enable connection-level live-to-live failover, but there's currently no ETA for that.

[jbertram]

Did you get your issue sorted out?

[raivil]

The team and I have done some more testing, and interestingly, in some random moments, the application did a live-live failover when a server went down. But it happened randomly. Very strange. We'll develop our own solution for now.

About the message distribution over networked nodes, we adjusted the connectors/acceptors configuration, multicast, etc and managed it to work correctly.

Thanks for the help.