-
1. Re: Cluster behavior - How deep is blue?
mcleanl Mar 24, 2010 4:04 PM (in response to blublinsky)Hi Boris,
It is very difficult to help you from what you have written. I have the following mental picture (which could be wrong)
Server1- Consumer
Contains an embedded HornetQ server instance running in cluster mode.
Runs an application that consumes messages from a specific queue
Server2- Producer
Contains an embedded HornetQ server instance running in cluster mode
Runs an application that produces messages to a specific queue
Is this correct? You might like to include your configuration files for the two servers.
-
2. Re: Cluster behavior - How deep is blue?
blublinsky Mar 24, 2010 5:30 PM (in response to mcleanl)This is correct.
I am also attaching zip file containing eclipse projects for both consumer and producer
-
Hornet.zip 3.7 MB
-
-
3. Re: Cluster behavior - How deep is blue?
clebert.suconic Mar 24, 2010 6:10 PM (in response to blublinsky)Well...
That means I will have to "debug" your code just to understand what you're saying and what might not be an issue in the end.
English is my second language (I originally speak Portuguese) but I still prefer reading English than java.
Can you please explain what you're trying to achieve better.. and if we then identify an issue we go to your code to replicate issues.
-
4. Re: Cluster behavior - How deep is blue?
blublinsky Mar 24, 2010 7:50 PM (in response to clebert.suconic)Lets me try again (English is my second language as well, so I prefer code):
Now that we know what two applications I have, here are two scenarios that I tried:
- Consumer first - works correctly:
- Start consumer
- Start Producer - join the cluster
- Producer send messages to queue
- Cluster recognize that this queue is defined on consumer and there is an active listener
- Cluster forwards messages to consumer
- Consumer reads messages
- Start consumer
- Producer first - fails
- Start producer
- Producer send messages to queue
- Start consumer - join the cluster
- Nothing happens
- Start producer
I have to guess, the problem is that if a server is part of a cluster, it distributes requests to the nodes for which, there is an active listener to a given queue. On another hand, if server has messages in the queue and additional listener is started, while messages are already in the queue, you are not trying to redistribute them.
- Consumer first - works correctly:
-
5. Re: Cluster behavior - How deep is blue?
timfox Mar 25, 2010 4:20 AM (in response to blublinsky)1 of 1 people found this helpfulBoris Lublinsky wrote:
I have to guess, the problem is that if a server is part of a cluster, it distributes requests to the nodes for which, there is an active listener to a given queue. On another hand, if server has messages in the queue and additional listener is started, while messages are already in the queue, you are not trying to redistribute them.
Yes, that's the default behaviour, this is discussed in the user manual.
http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html/clusters.html
In particular look at the chapters on Server Side Message Load-Balancing http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html/clusters.html#d0e9040
and message redistribution: http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html/clusters.html#clusters.message-redistribution
-
6. Re: Cluster behavior - How deep is blue?
blublinsky Mar 25, 2010 9:20 AM (in response to timfox)Thank you veru much Tim.
I re read redistribution piece, and according to documentation, it kicks in only if a client disconnects.
Do you have any plans to do redistribution on new server entering cluster?
Can you suggest any workaround, which will emulate this behavior?
-
7. Re: Cluster behavior - How deep is blue?
blublinsky Mar 25, 2010 10:17 AM (in response to blublinsky)The only solution, that I came up to solve this issue is to implement something like:
ConnectionFactory cf = HornetQJMSClient.createConnectionFactory(
new TransportConfiguration(InVMConnectorFactory.class.getName()));
Queue queue = new HornetQQueue(qname);
while(true){
// Take a break
Thread.sleep(5000);
Connection connection = null;
try{
connection = cf.createConnection();
connection.start();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
MessageConsumer messageConsumer = session.createConsumer(queue);
connection.start();
}
finally{
if (connection != null)
connection.close();
}
}In the separate thread, running in the message producer. This will force message redistribution. Is there any programmatic hook to the discovery? Rather then running it on the timer, I would much rather do it on event.
-
8. Re: Cluster behavior - How deep is blue?
timfox Mar 25, 2010 10:24 AM (in response to blublinsky)Boris Lublinsky wrote:
Thank you veru much Tim.
I re read redistribution piece, and according to documentation, it kicks in only if a client disconnects.
Do you have any plans to do redistribution on new server entering cluster?
Can you suggest any workaround, which will emulate this behavior?
You shouldn't need any work around.
If you have two nodes in the cluster: A and B. You have no consumers on node A, you have one message on node A. You create a consumer on node B, the message should get redistributed from A to B.
You must enable message redistribution for this to work.
-
9. Re: Cluster behavior - How deep is blue?
timfox Mar 25, 2010 10:27 AM (in response to timfox)There's a test for this case in the test suite.
testRedistributionWhenRemoteConsumerIsAdded
If that's testing what you describe, then I guess I haven't understood your explanation.
-
10. Re: Cluster behavior - How deep is blue?
blublinsky Mar 27, 2010 11:59 AM (in response to timfox)Tim,
this is not the scenario I anm looking for.
I have node A that has no consumer and is writing messages to the queue Q. Now node B is joining the cluster ans creates a listener on queue Q.
In this case messages sitting on queue Q on node A are not redistribured. The discovery event listener will add a new binding to the queue, but this does not cause redistribution.
The other thing is timimg.
A server start up returns back before a server joins the cluster, so doing cluster related things is not trivial.
Will it make sense to allow custom discovery listeners (for both adding and removing nodes) so that customer can write their extensions?
-
11. Re: Cluster behavior - How deep is blue?
timfox Mar 27, 2010 12:50 PM (in response to blublinsky)Can I verify that you have enabled message redistribution?
The scenario you describe should work. If it doesn't please file a JIRA and we will fix it.
-
12. Re: Cluster behavior - How deep is blue?
blublinsky Mar 27, 2010 10:05 PM (in response to timfox)Tim, here is my address-setting from hornetq-configuration
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>-1</max-size-bytes>
<page-size-bytes>10485760</page-size-bytes>
<redistribution-delay>10</redistribution-delay>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
</address-setting>I assume this is it. Do I need any other settings for redistribution?
-
13. Re: Cluster behavior - How deep is blue?
timfox Apr 12, 2010 3:24 AM (in response to blublinsky)That looks fine.
Boris - please file a JIRA with instructions to replicate etc, if this continues to be an issue.