HornetQ cluster questions

[zzz_public]

Hallo, dear HornetQ authors

I’m trying to do some tests with HornetQ cluster and it seems like there is a problem with my cluster installation. I’ve done almost default settings but it behaves like a set of independent servers.

• When I create a queue (durable or not) being connected to one node does HornetQ have to create such queues in other nodes?
• When there are queue named A instances on all cluster nodes and client producer connected to some node sends a message to A does a consumer connected to other node have to receive it? (I already know about redistribution-delay).

Let me say something about a configuration:

• hornetq-2.4.0.Final
• single HW machine running Windows 7, JDK 1.7.0_79
• cluster is made up of copies of hornetq-2.4.0.Final directory running on different ports
• I user core.api and does not plan to use JMS.

  I’ve attached all configs and logs. Here is something I think must be figured out explicitly: 

1. I clear all data content before all tests in run_standalone_clustered.bat (rm -rf %HORNETQ_HOME%\data\*) (rm works thankfully to UnxUtils)
2. I’ve deleted hornetq-jms.xml files from config\stand-alone\clustered for all nodes since I don’t use JMS.
3. I’ve added shared-store=false and cluster-user settings
4. My cluster-connection.address is “jms” and my queue name starts from “jms.”
5. I’ve added max-hops=1 which didn’t help
6. I have an address-setting match="#" with redistribution-delay = 0
7. I didn’t change hornetq-beans.xml, jndi.properties and switched most loggers in logging.properties to DEBUG.
8. Logs are situated in bin\logs\hornetq.log but it's not an issue for now.

I can see that nodes see each other but they don’t distribute queues and messages.

Could you help me please with my problems?

Best regards, Andrey

[zzz_public]

I've investigated MBeans and found that org.hornetq.Server.Core.BridgeNames is empty:

Name	Value	Type	Display Name	Update Interval	Description
BridgeNames	String[0]	String[]	BridgeNames	-1	Attribute exposed for management

I don't think it's correct but I cannot find a reason why it is so.

I've already tried static discovery:

   <connectors>     

      <connector name="netty">

         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

         <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>

      </connector>

      <!-- connector to the node01 -->

      <connector name="node01-connector">

         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

         <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

         <param key="port" value="11445"/>

      </connector>

   </connectors>

...

   <cluster-connections>

      <cluster-connection name="my-cluster">

         <address>jms</address> 

         <connector-ref>netty</connector-ref>

         <forward-when-no-consumers>true</forward-when-no-consumers>

         <max-hops>1</max-hops>

         <static-connectors allow-direct-connections-only="true">

            <connector-ref>node01-connector</connector-ref>

         </static-connectors>

      <!--discovery-group-ref discovery-group-name="dg-group1"/-->

      </cluster-connection>

   </cluster-connections>

Plus symmetric changes on node 01.

With no new results :-(

[zzz_public]

In fact server creates BridgeImpl instances, starts them and even add them into ClusterConnectionImpl.records. OrderedExecutorFactory does this by calling Topology.sendMemberUp with ClusterConnectionImpl registered as a listener. But nobody registers them in a resource manager.

[zzz_public]

I have found that hornetq-beans.xml contained ...bean name="StandaloneServer" class="org.hornetq.jms.server.impl.StandaloneNamingServer... which I don't like and also JMS server which I don't plan to use. Here is a simplifies hornetq-beans.xml.

<?xml version="1.0" encoding="UTF-8"?>

<deployment xmlns="urn:jboss:bean-deployer:2.0">

  <!-- MBean server -->

  <bean name="MBeanServer" class="javax.management.MBeanServer">

      <constructor factoryClass="java.lang.management.ManagementFactory"

                  factoryMethod="getPlatformMBeanServer"/>

  </bean>

  <!-- The core configuration -->

  <bean name="Configuration" class="org.hornetq.core.config.impl.FileConfiguration">

  </bean>

    <!-- The security manager -->

  <bean name="HornetQSecurityManager" class="org.hornetq.spi.core.security.HornetQSecurityManagerImpl">

      <start ignored="true"/>

      <stop ignored="true"/>

  </bean>

    <!-- The core server -->

  <bean name="HornetQServer" class="org.hornetq.core.server.impl.HornetQServerImpl">

      <constructor>

        <parameter>

            <inject bean="Configuration"/>

        </parameter>

        <parameter>

            <inject bean="MBeanServer"/>

        </parameter>

        <parameter>

            <inject bean="HornetQSecurityManager"/>

        </parameter>

      </constructor>

    <!--start ignored="true"/>

      <stop ignored="true"/-->

  </bean>

    <!-- The Stand alone server that controls the jndi server-->

  <!--bean name="StandaloneServer" class="org.hornetq.jms.server.impl.StandaloneNamingServer">

      <constructor>

        <parameter>

            <inject bean="HornetQServer"/>

        </parameter>

      </constructor>

      <property name="port">${jnp.port:1099}</property>

      <property name="bindAddress">${jnp.host:localhost}</property>

      <property name="rmiPort">${jnp.rmiPort:1098}</property>

      <property name="rmiBindAddress">${jnp.host:localhost}</property>

  </bean-->

  <!-- The JMS server -->

  <!--bean name="JMSServerManager" class="org.hornetq.jms.server.impl.JMSServerManagerImpl">

      <constructor>   

        <parameter>

            <inject bean="HornetQServer"/>

        </parameter>   

      </constructor>

  </bean-->

</deployment>

But it still does not forward messages from one not to another. Even whan I precreate a queue.

[zzz_public]

Thenks to everybody who read this.

I have had interpated forward-when-no-consumers key incorrectly. It forces a cluster to round robin messages preventing it from delivering messages to existing consumers. I'm not sure about the use case when this flag must be true. This means that we believe that all cluster nodes will have consumers and if some will not a subset of messages will in fact be ignored by the system.

Anyway <forward-when-no-consumers>false</forward-when-no-consumers> is th solution for the question "When there are queue named A instances on all cluster nodes and client producer connected to some node sends a message to A does a consumer connected to other node have to receive it?"

Lets investigate the first one.

[zzz_public]

Notes regarding 1-st question:

• When I create a queue (durable or not) being connected to one node does HornetQ have to create such queues in other nodes?

There are no code in examples which creates queue programmatically via JMS or core API. When I use coreSession.createQueue(getQueueName(), getQueueName(), DURABLE); it creates queue just on a node coreSession connected to. But other nodes does not have the queue.

I believe client must not deal with such issues. It must create a queue on a cluster as a whole. What will happen if client that was created a queue on all nodes disconnects and a new node comes?

[zzz_public]

Currently I'm sure that HornetQ server does not automatically maintain a consistent set of queues over cluster nodes.

Can anyone tell how to get topology listener to get topology changes to maintain the queues set myself?

I've tried to HornetQServer.getHornetQServerControl().addNotificationListener(...) - nobody calls my listener!

I know the discovery configuration and I've tried:

            ServerLocatorInternal locator = (ServerLocatorInternal) HornetQClient.createServerLocatorWithHA(configBuilder.getDiscoveryGroupConfiguration());

            locator.addClusterTopologyListener(

                    new org.hornetq.api.core.client.ClusterTopologyListener() {

                        @Override

                        public void nodeUP(TopologyMember member, boolean last) {

                            ...

                        }

                        @Override

                        public void nodeDown(long eventUID, String nodeID) {

                            ...

                        }

                    });

Nobody calls my listener!

[zzz_public]

I've found a solution. I register my incoming Interceptor, filter topology related packets and work them up. Unfortunately, one may only call server.getRemotingService().addIncomingInterceptor(...) only after server has already started and some packets may have already been missed. I can add initial topology discovery manually after listener was registered or add an interceptor via a configuration. The second way is not so handy since one cannot provide an interceptor instance but just a class name.

I wonder does anyone use HornetQ as a cluster without knowing about all queues in advance? Am I the only damned person?

[jbertram]

As I understand it, HornetQ will actually maintain a consistent set of queues over all the cluster nodes.  What specifically are you observing which leads you to believe that it doesn't?

Also, the topology listener is not for queues but for entire nodes coming and going from the cluster (hence the nodeUP and nodeDown methods).

[jbertram]

I'm still not clear on exactly what you're trying to accomplish and what exact problems you're experiencing.  Can you clarify this?

[zzz_public]

Thanks a lot for your reply!

I have done some tests which briefly do following:

1. I run 2 (or more) nodes. I see in logs that both nodes see each other.

2. A test client:

- connects to node 1, creates a queue and sends a message to it.

- connects to node 2 an tries to receive a message.

But there is no queue in node 2.

I've attached a test to the root message of this conversation - ClusterClientHornetQCoreTest.java.

[zzz_public]

I just want my cluster to act as an opaque service from clients point of view. I've tested that if all nodes have a queue then client A may send a message and client B will receive it regardless of the node he is connected to. It's great!

But I don't know the set of queues at compile time. In fact my clients may create and delete them at runtime. And I have found that when client A creates a queue Q this queue remains on a just a node A was connected to. I have investigated it using test clients and JMS interface. May be I have to do some cluster settings for HornetQ to be able to distribute queues?

I use hornetq-configuration.xml attached to this message.

[zzz_public]

>> Also, the topology listener is not for queues but for entire nodes coming and going from the cluster (hence the nodeUP and nodeDown methods).

Yes, I know that. I tried to get topology notifications and queue creation and deletion events to incorporate some form of consensus algorithm to maintain a consistent queues set on all nodes myself.

[jbertram]

Can you attach the logs from the 2 nodes when you run your test?  My hunch is that your nodes are not actually clustering properly.

Also, are you running your HornetQ instances on separate machines or the same machine?

Lastly, are you passing any system properties when you start HornetQ?  If so, what are they properties and values which you're passing?

[zzz_public]

Thank you for your answers.

Files are:

adm_20150723_144842.zip - almost full directories content with logs, journals, configs and scripts. I've just removed jars to reduce file size.

hornetq00.log - is a log of node 00. It also could be found in adm_20150723_144842.zip\adm\hornetq-node00\bin\logs\hornetq.log

hornetq01.log - is a log of node 01. It also could be found in adm_20150723_144842.zip\adm\hornetq-node01\bin\logs\hornetq.log

ClusterClientHornetQCoreTest.java - is a code of my test.

test output.txt - test output.

I run instances on a single hardware machine but on different ports and in different dirs.

System properties was taken from a sample they can be found in adm_20150723_144842.zip\adm\hornetq-node0x\bin\run_standalone_clustered.bat - nothing special, just port number.

And I also edited hornetq-beans.xml files as I mentioned previously. It can be found in adm_20150723_144842.zip\adm\hornetq-node0x\config\stand-alone\clustered\hornetq-beans.xml

Just to clarity the target. I need a symmetric cluster.