Cluster behavior

[blublinsky]

I am trying to build a peer-to-peer cluster. In a simplle example I created one application that runs embedded server and reads messages from a given queue - consumer and the other one, which runs embedded server and writes messages to the queue - producer.

If a start consumer and then producer, it works great. If the startup order is different, consumer does not pick up messages. It looks like, messages that are already inserted in producer's queue are not delivered to the clustered node when it is inserted.

Am I missing something?

[mcleanl]

Hi Boris,

It is very difficult to help you from what you have written.  I have the following mental picture (which could be wrong)

Server1- Consumer

Contains an embedded HornetQ server instance running in cluster mode.

Runs an application that consumes messages from a specific queue

Server2- Producer

Contains an embedded HornetQ server instance running in cluster mode

Runs an application that produces messages to a specific queue

Is this correct?  You might like to include your configuration files for the two servers.

[blublinsky]

This is correct.

I am also attaching zip file containing eclipse projects for both consumer and producer

[clebert.suconic]

Well...

That means I will have to "debug" your code just to understand what you're saying and what might not be an issue in the end.

English is my second language (I originally speak Portuguese) but I still prefer reading English than java.

Can you please explain what you're trying to achieve better.. and if we then identify an issue we go to your code to replicate issues.

[blublinsky]

Lets me try again (English is my second language as well, so I prefer code):

Now that we know what two applications I have, here are two scenarios that I tried:

1. Consumer first - works correctly:
   
   1. Start consumer
      
   2. Start Producer - join the cluster
      
   3. Producer send messages to queue
      
   4. Cluster recognize that this queue is defined on consumer and there is an active listener
      
   5. Cluster forwards messages to consumer
      
   6. Consumer reads messages
      
2. Producer first - fails
   
   1. Start producer
      
   2. Producer send messages to queue
      
   3. Start consumer - join the cluster
      
   4. Nothing happens
      

I have to guess, the problem is that if a server is part of a cluster, it distributes requests to the nodes for which, there is an active listener to a given queue. On another hand, if server has messages in the queue and additional listener is started, while messages are already in the queue, you are not trying to redistribute them.

[timfox]

Boris Lublinsky wrote:

I have to guess, the problem is that if a server is part of a cluster, it distributes requests to the nodes for which, there is an active listener to a given queue. On another hand, if server has messages in the queue and additional listener is started, while messages are already in the queue, you are not trying to redistribute them.

Yes, that's the default behaviour, this is discussed in the user manual.

http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html/clusters.html

In particular look at the chapters on Server Side Message Load-Balancing http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html/clusters.html#d0e9040

and message redistribution: http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html/clusters.html#clusters.message-redistribution

[blublinsky]

Thank you veru much Tim.

I re read redistribution piece, and according to documentation, it kicks in only if a client disconnects.

Do you have any plans to do redistribution on new server entering cluster?

Can you suggest any workaround, which will emulate this behavior?

[blublinsky]

The only solution, that I came up to solve this issue is to implement  something like:

     ConnectionFactory cf =  HornetQJMSClient.createConnectionFactory(
                 new  TransportConfiguration(InVMConnectorFactory.class.getName()));
         Queue queue = new HornetQQueue(qname);
         while(true){
             // Take a break
             Thread.sleep(5000);
             Connection connection = null;
             try{
                 connection = cf.createConnection();
                 connection.start();
                 Session session = connection.createSession(false,  Session.AUTO_ACKNOWLEDGE);
                 MessageConsumer messageConsumer =  session.createConsumer(queue);
                 connection.start();
              }
             finally{
                 if (connection != null)
                     connection.close();
             }
         }

In the separate thread, running in the message  producer. This will force message redistribution. Is there any  programmatic hook to the discovery? Rather then running it on the timer,  I would much rather do it on event.

[timfox]

Boris Lublinsky wrote:

Thank you veru much Tim.

I re read redistribution piece, and according to documentation, it kicks in only if a client disconnects.

Do you have any plans to do redistribution on new server entering cluster?

Can you suggest any workaround, which will emulate this behavior?

You shouldn't need any work around.

If you have two nodes in the cluster: A and B. You have no consumers on node A, you have one message on node A. You create a consumer on node B, the message should get redistributed from A to B.

You must enable message redistribution for this to work.

[timfox]

There's a test for this case in the test suite.

See http://anonsvn.jboss.org/repos/hornetq/tags/HornetQ_2_0_0_GA/tests/src/org/hornetq/tests/integration/cluster/distribution/MessageRedistributionTest.java

testRedistributionWhenRemoteConsumerIsAdded

If that's testing what you describe, then I guess I haven't understood your explanation.

[blublinsky]

Tim,

this is not the scenario I anm looking for.

I have node A that has no consumer and is writing messages to the queue Q. Now node B is joining the cluster ans creates a listener on queue Q.

In this case messages sitting on queue Q on node A are not redistribured. The discovery event listener will add a new binding to the queue, but this does not cause redistribution.

The other thing is timimg.

A server start up returns back before a server joins the cluster, so doing cluster related things is not trivial.

Will it make sense to allow custom discovery listeners (for both adding and removing nodes) so that customer can write their extensions?

[timfox]

Can I verify that you have enabled message redistribution?

The scenario you describe should work. If it doesn't please file a JIRA and we will fix it.

[blublinsky]

Tim, here is my address-setting from hornetq-configuration

     <address-setting match="#">
            <dead-letter-address>jms.queue.DLQ</dead-letter-address>
            <expiry-address>jms.queue.ExpiryQueue</expiry-address>
            <redelivery-delay>0</redelivery-delay>
            <max-size-bytes>-1</max-size-bytes>
            <page-size-bytes>10485760</page-size-bytes>
            <redistribution-delay>10</redistribution-delay>
            <message-counter-history-day-limit>10</message-counter-history-day-limit>
        </address-setting>

I assume this is it. Do I need any other settings for redistribution?

[timfox]

That looks fine.

Boris - please file a JIRA with instructions to replicate etc, if this continues to be an issue.

[timfox]

https://jira.jboss.org/jira/browse/HORNETQ-359