1 2 Previous Next 21 Replies Latest reply on Jul 27, 2015 6:00 PM by Clebert Suconic

    HornetQ cluster questions

    Andrey zzz_public Newbie

      Hallo, dear HornetQ authors

       

      I’m trying to do some tests with HornetQ cluster and it seems like there is a problem with my cluster installation. I’ve done almost default settings but it behaves like a set of independent servers.

      • When I create a queue (durable or not) being connected to one node does HornetQ have to create such queues in other nodes?
      • When there are queue named A instances on all cluster nodes and client producer connected to some node sends a message to A does a consumer connected to other node have to receive it? (I already know about redistribution-delay).

      Let me say something about a configuration:

      • hornetq-2.4.0.Final
      • single HW machine running Windows 7, JDK 1.7.0_79
      • cluster is made up of copies of hornetq-2.4.0.Final directory running on different ports
      • I user core.api and does not plan to use JMS.

        I’ve attached all configs and logs. Here is something I think must be figured out explicitly: 

        1. I clear all data content before all tests in run_standalone_clustered.bat (rm -rf %HORNETQ_HOME%\data\*) (rm works thankfully to UnxUtils)
        2. I’ve deleted hornetq-jms.xml files from config\stand-alone\clustered for all nodes since I don’t use JMS.
        3. I’ve added shared-store=false and cluster-user settings
        4. My cluster-connection.address is “jms” and my queue name starts from “jms.”
        5. I’ve added max-hops=1 which didn’t help
        6. I have an address-setting match="#" with redistribution-delay = 0
        7. I didn’t change hornetq-beans.xml, jndi.properties and switched most loggers in logging.properties to DEBUG.
        8. Logs are situated in bin\logs\hornetq.log but it's not an issue for now.

       

      I can see that nodes see each other but they don’t distribute queues and messages.

      Could you help me please with my problems?

       

      Best regards, Andrey

        • 1. Re: HornetQ cluster questions
          Andrey zzz_public Newbie

          I've investigated MBeans and found that org.hornetq.Server.Core.BridgeNames is empty:

          NameValueTypeDisplay NameUpdate IntervalDescription
          BridgeNamesString[0]String[]BridgeNames-1Attribute exposed for management

          I don't think it's correct but I cannot find a reason why it is so.

          I've already tried static discovery:

             <connectors>     

                <connector name="netty">

                   <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                   <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

                   <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>

                </connector>

                <!-- connector to the node01 -->

                <connector name="node01-connector">

                   <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                   <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

                   <param key="port" value="11445"/>

                </connector>

             </connectors>

          ...

             <cluster-connections>

                <cluster-connection name="my-cluster">

                   <address>jms</address> 

                   <connector-ref>netty</connector-ref>

                   <forward-when-no-consumers>true</forward-when-no-consumers>

                   <max-hops>1</max-hops>

                   <static-connectors allow-direct-connections-only="true">

                      <connector-ref>node01-connector</connector-ref>

                   </static-connectors>

                <!--discovery-group-ref discovery-group-name="dg-group1"/-->

                </cluster-connection>

             </cluster-connections>

          Plus symmetric changes on node 01.

          With no new results :-(

          • 2. Re: HornetQ cluster questions
            Andrey zzz_public Newbie

            In fact server creates BridgeImpl instances, starts them and even add them into ClusterConnectionImpl.records. OrderedExecutorFactory does this by calling Topology.sendMemberUp with ClusterConnectionImpl registered as a listener. But nobody registers them in a resource manager.

            • 3. Re: HornetQ cluster questions
              Andrey zzz_public Newbie

              I have found that hornetq-beans.xml contained ...bean name="StandaloneServer" class="org.hornetq.jms.server.impl.StandaloneNamingServer... which I don't like and also JMS server which I don't plan to use. Here is a simplifies hornetq-beans.xml.

              <?xml version="1.0" encoding="UTF-8"?>

              <deployment xmlns="urn:jboss:bean-deployer:2.0">

                <!-- MBean server -->

                <bean name="MBeanServer" class="javax.management.MBeanServer">

                    <constructor factoryClass="java.lang.management.ManagementFactory"

                                factoryMethod="getPlatformMBeanServer"/>

                </bean>

               

                <!-- The core configuration -->

                <bean name="Configuration" class="org.hornetq.core.config.impl.FileConfiguration">

                </bean>

               

                  <!-- The security manager -->

                <bean name="HornetQSecurityManager" class="org.hornetq.spi.core.security.HornetQSecurityManagerImpl">

                    <start ignored="true"/>

                    <stop ignored="true"/>

                </bean>

               

                  <!-- The core server -->

                <bean name="HornetQServer" class="org.hornetq.core.server.impl.HornetQServerImpl">

                    <constructor>

                      <parameter>

                          <inject bean="Configuration"/>

                      </parameter>

                      <parameter>

                          <inject bean="MBeanServer"/>

                      </parameter>

                      <parameter>

                          <inject bean="HornetQSecurityManager"/>

                      </parameter>

                    </constructor>

                  <!--start ignored="true"/>

                    <stop ignored="true"/-->

                </bean>

               

                  <!-- The Stand alone server that controls the jndi server-->

                <!--bean name="StandaloneServer" class="org.hornetq.jms.server.impl.StandaloneNamingServer">

                    <constructor>

                      <parameter>

                          <inject bean="HornetQServer"/>

                      </parameter>

                    </constructor>

                    <property name="port">${jnp.port:1099}</property>

                    <property name="bindAddress">${jnp.host:localhost}</property>

                    <property name="rmiPort">${jnp.rmiPort:1098}</property>

                    <property name="rmiBindAddress">${jnp.host:localhost}</property>

                </bean-->

               

                <!-- The JMS server -->

                <!--bean name="JMSServerManager" class="org.hornetq.jms.server.impl.JMSServerManagerImpl">

                    <constructor>   

                      <parameter>

                          <inject bean="HornetQServer"/>

                      </parameter>   

                    </constructor>

                </bean-->

              </deployment>

               

              But it still does not forward messages from one not to another. Even whan I precreate a queue.

              • 4. Re: HornetQ cluster questions
                Andrey zzz_public Newbie

                Thenks to everybody who read this.

                I have had interpated forward-when-no-consumers key incorrectly. It forces a cluster to round robin messages preventing it from delivering messages to existing consumers. I'm not sure about the use case when this flag must be true. This means that we believe that all cluster nodes will have consumers and if some will not a subset of messages will in fact be ignored by the system.

                Anyway <forward-when-no-consumers>false</forward-when-no-consumers> is th solution for the question "When there are queue named A instances on all cluster nodes and client producer connected to some node sends a message to A does a consumer connected to other node have to receive it?"

                Lets investigate the first one.

                1 of 1 people found this helpful
                • 5. Re: HornetQ cluster questions
                  Andrey zzz_public Newbie

                  Notes regarding 1-st question:

                  • When I create a queue (durable or not) being connected to one node does HornetQ have to create such queues in other nodes?

                  There are no code in examples which creates queue programmatically via JMS or core API. When I use coreSession.createQueue(getQueueName(), getQueueName(), DURABLE); it creates queue just on a node coreSession connected to. But other nodes does not have the queue.

                  I believe client must not deal with such issues. It must create a queue on a cluster as a whole. What will happen if client that was created a queue on all nodes disconnects and a new node comes?

                  • 6. Re: HornetQ cluster questions
                    Andrey zzz_public Newbie

                    Currently I'm sure that HornetQ server does not automatically maintain a consistent set of queues over cluster nodes.

                    Can anyone tell how to get topology listener to get topology changes to maintain the queues set myself?

                    I've tried to HornetQServer.getHornetQServerControl().addNotificationListener(...) - nobody calls my listener!

                     

                     

                    I know the discovery configuration and I've tried:

                                ServerLocatorInternal locator = (ServerLocatorInternal) HornetQClient.createServerLocatorWithHA(configBuilder.getDiscoveryGroupConfiguration());

                                locator.addClusterTopologyListener(

                                        new org.hornetq.api.core.client.ClusterTopologyListener() {

                                            @Override

                                            public void nodeUP(TopologyMember member, boolean last) {

                                                ...

                                            }

                                            @Override

                                            public void nodeDown(long eventUID, String nodeID) {

                                                ...

                                            }

                                        });

                    Nobody calls my listener!

                    • 7. Re: HornetQ cluster questions
                      Andrey zzz_public Newbie

                      I've found a solution. I register my incoming Interceptor, filter topology related packets and work them up. Unfortunately, one may only call server.getRemotingService().addIncomingInterceptor(...) only after server has already started and some packets may have already been missed. I can add initial topology discovery manually after listener was registered or add an interceptor via a configuration. The second way is not so handy since one cannot provide an interceptor instance but just a class name.

                      I wonder does anyone use HornetQ as a cluster without knowing about all queues in advance? Am I the only damned person?

                      • 8. Re: HornetQ cluster questions
                        Justin Bertram Master

                        As I understand it, HornetQ will actually maintain a consistent set of queues over all the cluster nodes.  What specifically are you observing which leads you to believe that it doesn't?

                         

                        Also, the topology listener is not for queues but for entire nodes coming and going from the cluster (hence the nodeUP and nodeDown methods).

                        • 9. Re: HornetQ cluster questions
                          Justin Bertram Master

                          I'm still not clear on exactly what you're trying to accomplish and what exact problems you're experiencing.  Can you clarify this?

                          • 10. Re: HornetQ cluster questions
                            Andrey zzz_public Newbie

                            Thanks a lot for your reply!

                            I have done some tests which briefly do following:

                            1. I run 2 (or more) nodes. I see in logs that both nodes see each other.

                            2. A test client:

                            - connects to node 1, creates a queue and sends a message to it.

                            - connects to node 2 an tries to receive a message.

                            But there is no queue in node 2.

                             

                            I've attached a test to the root message of this conversation - ClusterClientHornetQCoreTest.java.

                            • 11. Re: HornetQ cluster questions
                              Andrey zzz_public Newbie

                              I just want my cluster to act as an opaque service from clients point of view. I've tested that if all nodes have a queue then client A may send a message and client B will receive it regardless of the node he is connected to. It's great!

                              But I don't know the set of queues at compile time. In fact my clients may create and delete them at runtime. And I have found that when client A creates a queue Q this queue remains on a just a node A was connected to. I have investigated it using test clients and JMS interface. May be I have to do some cluster settings for HornetQ to be able to distribute queues?

                              I use hornetq-configuration.xml attached to this message.

                              • 12. Re: HornetQ cluster questions
                                Andrey zzz_public Newbie

                                >> Also, the topology listener is not for queues but for entire nodes coming and going from the cluster (hence the nodeUP and nodeDown methods).

                                Yes, I know that. I tried to get topology notifications and queue creation and deletion events to incorporate some form of consensus algorithm to maintain a consistent queues set on all nodes myself.

                                • 13. Re: HornetQ cluster questions
                                  Justin Bertram Master

                                  Can you attach the logs from the 2 nodes when you run your test?  My hunch is that your nodes are not actually clustering properly.

                                   

                                  Also, are you running your HornetQ instances on separate machines or the same machine?

                                   

                                  Lastly, are you passing any system properties when you start HornetQ?  If so, what are they properties and values which you're passing?

                                  • 14. Re: HornetQ cluster questions
                                    Andrey zzz_public Newbie

                                    Thank you for your answers.

                                    Files are:

                                    adm_20150723_144842.zip - almost full directories content with logs, journals, configs and scripts. I've just removed jars to reduce file size.

                                    hornetq00.log - is a log of node 00. It also could be found in adm_20150723_144842.zip\adm\hornetq-node00\bin\logs\hornetq.log

                                    hornetq01.log - is a log of node 01. It also could be found in adm_20150723_144842.zip\adm\hornetq-node01\bin\logs\hornetq.log

                                    ClusterClientHornetQCoreTest.java - is a code of my test.

                                    test output.txt - test output.

                                     

                                    I run instances on a single hardware machine but on different ports and in different dirs.

                                     

                                    System properties was taken from a sample they can be found in adm_20150723_144842.zip\adm\hornetq-node0x\bin\run_standalone_clustered.bat - nothing special, just port number.


                                    And I also edited hornetq-beans.xml files as I mentioned previously. It can be found in adm_20150723_144842.zip\adm\hornetq-node0x\config\stand-alone\clustered\hornetq-beans.xml


                                    Just to clarity the target. I need a symmetric cluster.

                                    1 2 Previous Next