-
1. Re: Message redistribution
timfox Feb 17, 2008 2:59 PM (in response to martin.wickus)Please can you be more specific about what you have done, and what you have observed (step by step instructions).
-
2. Re: Message redistribution
martin.wickus Feb 18, 2008 5:28 AM (in response to martin.wickus)Precondition: JBM cluster consisting of two nodes posting messages to a clustered queue. The JBM cluster is formed across JBoss nodes.
Step1. Start up a plain java process (configured with JBM client libraries and patched JBoss Remoting lib) which looks up a ClusteredConnectionFactory from the JBoss/JBM partition via JNDI, creates a connection then registers a MessageListener to receive messages. Let's say the resulting connection is attached to node 1 of the JBM cluster.
Step2. Send a message from node 1 of the JBM cluster to the clustered queue. The message is immediately received by the message listener.
Step3. Send a message from node 2 of the JBM cluster. The message is not received by the message listener.
Step4. Kill the external java process hosting the MessageListener.
Step5. Start the external java process hosting the MessageListener. This is a repeat of Step1, but this time my connection is to node 2. Backlog of messages sent from node 2 are immediately received.
Step6. Send a message from node 2 of the JBM cluster to the clustered queue. The message is immediately received by the message listener.
Step7. Send a message from node 1 of the JBM cluster. The message is not received by the message listener.
Red Hat JBoss support has let me know that a consumer on node 1 is able to consume messages on the same distributed queue hosted on node 2. However, the messages will not be "sucked" over until the local queue is empty. So, for example, if Consumer A is connected to node 1, messages in the distributed queue on node 2 will not be moved to node 1 until Consumer A has exhausted all messages in the queue on node 1. That is, messages queued to node 1 would have precedence over messages queued to node 2 for Consumer A.
Two things from this then:
1. I am not seeing the behavior described by Red Hat (messages seem to stay stuck in their original node).
2. The behavior described by Red Hat would mean that if any activity is generated by node 1 (eg. even just 1 message on the distributed queue for node 1) then none of the messages on node 2 will get delivered since they won't be sucked over to node 1.
I'd prefer being able to look at the distributed queue as a single virtual queue from which my consumer can take messages without having to worry from where a message originated. -
3. Re: Message redistribution
timfox Feb 18, 2008 5:40 AM (in response to martin.wickus)"martin.wickus" wrote:
Precondition: JBM cluster consisting of two nodes posting messages to a clustered queue. The JBM cluster is formed across JBoss nodes.
Step1. Start up a plain java process (configured with JBM client libraries and patched JBoss Remoting lib) which looks up a ClusteredConnectionFactory from the JBoss/JBM partition via JNDI, creates a connection then registers a MessageListener to receive messages. Let's say the resulting connection is attached to node 1 of the JBM cluster.
Step2. Send a message from node 1 of the JBM cluster to the clustered queue. The message is immediately received by the message listener.
Step3. Send a message from node 2 of the JBM cluster. The message is not received by the message listener.
That should certainly work (it's the simplest possible redistribution scenario). Things to check:
1) You have deployed your queue as clustered
2) ClusterPullConnectionFactory is specified in the server peer config and is deployed
3) Something is wrong in your JGroups config.
Most likely is 3)
Take a look at the JGroups site for some tips on how to work out whether the nodes are seeing each other etc. -
4. Re: Message redistribution
timfox Feb 18, 2008 5:41 AM (in response to martin.wickus)Also make sure you don't have other consumers on the nodes. Any local consumers will always get the message if they are not busy.
-
5. Re: Message redistribution
timfox Feb 18, 2008 5:44 AM (in response to martin.wickus)Final thing to note:
In 95% of user deployments, message redistribution is NOT necessary.
Typically users deploy a homogeneous bank of MDBs (or other consumers) across the cluster - the same ones on each node. Client producing connections are also load balanced across the cluster. Since there's an even mix of producers/consumers across the cluster there's no need to distribute.
Distribution is uiseful in the case:
1) You have a bank of many consumers on a cluster but only, say., one or two consumers on a small number of nodes.
2) As above but swap producers with consumers -
6. Re: Message redistribution
timfox Feb 18, 2008 5:59 AM (in response to martin.wickus)There is also a simple distributed queue example in the distro that we recommend you run (see install guide) - can you see if this works?
-
7. Re: Message redistribution
martin.wickus Feb 18, 2008 6:23 AM (in response to martin.wickus)1)You have deployed your queue as clustered
2) ClusterPullConnectionFactory is specified in the server peer config and is deployed
3) Something is wrong in your JGroups config.
1. - confirmed (via JMX attribute of destination in JMX-Console)
2. - confirmed (JMX mbean present in JMX-Console)
3. - Might need to investigate more here. However, I do find JBM clustering working for another scenario. I have a distributed topic and when I publish from any node, the resulting messages are consumed by all MDBs in cluster (1 such MDB per node).
Will check the distributed example and report back here. -
8. Re: Message redistribution
martin.wickus Feb 18, 2008 7:16 AM (in response to martin.wickus)Disclaimer: I used my own queue (configured as per bottom of post)
I ran the test 5 times in a row:
1. Failure
2. Success
3. Success
4. Success
5. Failure
The two failures had the same error message:
C:\opt\downloads\jboss\jboss-messaging-1.4.0.SP3\jboss-messaging-1.4.0.SP3\examples\distributed-queue>ant -Djboss.home=C:\dev\abfo_home\DEV09\abf
o-1.2-clustering\abfo-jboss-1.2-clustering-dev-1 -Djboss.configuration0=abfo -Dexample.queue.name=OTL-LIFFEF-Send -Djava.naming.provider.url=jnp:
//10.142.204.77:59098ant -Djboss.home=C:\dev\abfo_home\DEV09\abfo-1.2-clustering\abfo-jboss-1.2-clustering-dev-1 -Djboss.configuration0=abfo -Dex
ample.queue.name=OTL-LIFFEF-Send
Buildfile: build.xml
identify:
[echo] ###########################################################################
[echo] # Running the DISTRIBUTED QUEUE example #
[echo] ###########################################################################
[echo] The queue: OTL-LIFFEF-Send
[echo] The client jar: ../..//jboss-messaging-client.jar
sanity-check:
init:
compile:
run:
[java] Distributed queue /queue/OTL-LIFFEF-Send exists
[java] The message was successfully sent to the distributed queue
[java] java.lang.NullPointerException
[java] at org.jboss.example.jms.distributedqueue.DistributedQueueExample.example(DistributedQueueExample.java:112)
[java] at org.jboss.example.jms.common.ExampleSupport.run(ExampleSupport.java:147)
[java] at org.jboss.example.jms.distributedqueue.DistributedQueueExample.main(DistributedQueueExample.java:167)
[java]
[java] #####################
[java] ### FAILURE! ###
[java] #####################
Queue configuration:
<!-- destinations for sending and receiving orders -->
<mbean code="org.jboss.jms.server.destination.QueueService"
name="jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send-DLQ"
xmbean-dd="xmdesc/Queue-xmbean.xml">
<depends optional-attribute-name="ServerPeer">
jboss.messaging:service=ServerPeer
jboss.messaging:service=PostOffice
true
<mbean code="org.jboss.jms.server.destination.QueueService"
name="jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send-ExpiryQueue"
xmbean-dd="xmdesc/Queue-xmbean.xml">
<depends optional-attribute-name="ServerPeer">
jboss.messaging:service=ServerPeer
jboss.messaging:service=PostOffice
true
<mbean code="org.jboss.jms.server.destination.QueueService"
name="jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send"
xmbean-dd="xmdesc/Queue-xmbean.xml">
<depends optional-attribute-name="ServerPeer">
jboss.messaging:service=ServerPeer
jboss.messaging:service=PostOffice
jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send-DLQ
jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send-ExpiryQueue
true -
9. Re: Message redistribution
timfox Feb 18, 2008 7:21 AM (in response to martin.wickus)Right - this implies that distribution is not working for you - it works about half the time, and about half the time the consumer will be created (randomly) on the same node.
Almost certainly there's something wrong in your JGroups setup -
10. Re: Message redistribution
timfox Feb 18, 2008 7:22 AM (in response to martin.wickus)BTW, can you please try with the preconfigured queues (i.e. do not change ANYTHING from the example) from the distro - i.e. don't use your own queue.
-
11. Re: Message redistribution
martin.wickus Feb 18, 2008 7:24 AM (in response to martin.wickus)<mbean code="org.jboss.jms.server.destination.QueueService" name="jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send" xmbean-dd="xmdesc/Queue-xmbean.xml"> <depends optional-attribute-name="ServerPeer"> jboss.messaging:service=ServerPeer </depends> <depends>jboss.messaging:service=PostOffice</depends> <attribute name="DLQ"> jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send-DLQ </attribute> <attribute name="ExpiryQueue"> jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send-ExpiryQueue </attribute> <attribute name="SecurityConfig"> <security> <role name="guest" read="true" write="true" /> </security> </attribute> <attribute name="Clustered">true</attribute> </mbean>
-
12. Re: Message redistribution
martin.wickus Feb 18, 2008 7:44 AM (in response to martin.wickus)I added the test destinations to my configurations and it works with the default test queue. This would indicate something is wrong with the way my queue is configured. I'm about to make the configuration more similiar (remove DLQ, ExpiryQueue and SecurityConfig)
-
13. Re: Message redistribution
martin.wickus Feb 18, 2008 9:13 AM (in response to martin.wickus)I got it working, but something strange is afoot here.
Although the JMX-Console showed my mbeans are clustered, the two entries in the table JBM_POSTOFFICE did NOT. In other words, the flag for the CLUSTERED column was set to 'N'.
I deleted all entries from the JBM_POSTOFFICE table and restarted my nodes. This time, the clustered flags were set correct in the database and everything worked as expected. I ran the distributed-queue example with my own queue and it worked 10 times out of 10.
I only recently clustered this destination, so I wonder if after making the config change, the entry didn't get updated in the database properly. I'll see if I can re-produce this and will report back. -
14. Re: Message redistribution
martin.wickus Feb 18, 2008 10:51 AM (in response to martin.wickus)I can confirm that updating a destinations' configuration (eg. setting the clustered attribute to true) doesn't seem to update the configuration in JBM_POSTOFFICE.