1 2 Previous Next 15 Replies Latest reply on Feb 18, 2008 11:42 AM by timfox

    Message redistribution

    martin.wickus

      We are running JBoss EAP 4.3 which comes as standard with JBoss Messaging 1.4.0.SP3.

      Two nodes are running in clustered configuration.

      Sending messages from nodes to distributed/clustered destination.

      External consumer (not in cluster or even in JBoss) consuming from distributed queue. Connection was obtained via ClusteredConnectionFactory.

      Problem: Only consuming messages from partial queue of node connected to.

      Elaboration:
      Each of the nodes send messages to the distributed queue. The messages obviously go into their respective partial queues.

      The external consumer will get a connection to one of the nodes (sticky once created) and hence can receive messages only from the partial queue on the node it is connected to. In the meantime, my messages are bundling up on the other partial queue. If I kill my receiver connection, the ClusteredConnectionFactory is smart enough to dish out a connection to the node with the backlog of messages. However, that means messages will now start backlogging on the other node.

      After reading Tim Fox’s explanation of the design behind JBM clustering, I understood the Intelligent Balancer would redistribute messages from one partial queue to another based on real-time statistics. I am not seeing this behaviour however, thus perhaps there are additional configuration changes I need to carry out.

      PS. JBoss Messaging User Guide doesn’t discuss the Intelligent Balancer or Message Redistribution.



        • 1. Re: Message redistribution
          timfox

          Please can you be more specific about what you have done, and what you have observed (step by step instructions).

          • 2. Re: Message redistribution
            martin.wickus

            Precondition: JBM cluster consisting of two nodes posting messages to a clustered queue. The JBM cluster is formed across JBoss nodes.

            Step1. Start up a plain java process (configured with JBM client libraries and patched JBoss Remoting lib) which looks up a ClusteredConnectionFactory from the JBoss/JBM partition via JNDI, creates a connection then registers a MessageListener to receive messages. Let's say the resulting connection is attached to node 1 of the JBM cluster.

            Step2. Send a message from node 1 of the JBM cluster to the clustered queue. The message is immediately received by the message listener.

            Step3. Send a message from node 2 of the JBM cluster. The message is not received by the message listener.

            Step4. Kill the external java process hosting the MessageListener.

            Step5. Start the external java process hosting the MessageListener. This is a repeat of Step1, but this time my connection is to node 2. Backlog of messages sent from node 2 are immediately received.

            Step6. Send a message from node 2 of the JBM cluster to the clustered queue. The message is immediately received by the message listener.

            Step7. Send a message from node 1 of the JBM cluster. The message is not received by the message listener.

            Red Hat JBoss support has let me know that a consumer on node 1 is able to consume messages on the same distributed queue hosted on node 2. However, the messages will not be "sucked" over until the local queue is empty. So, for example, if Consumer A is connected to node 1, messages in the distributed queue on node 2 will not be moved to node 1 until Consumer A has exhausted all messages in the queue on node 1. That is, messages queued to node 1 would have precedence over messages queued to node 2 for Consumer A.

            Two things from this then:

            1. I am not seeing the behavior described by Red Hat (messages seem to stay stuck in their original node).
            2. The behavior described by Red Hat would mean that if any activity is generated by node 1 (eg. even just 1 message on the distributed queue for node 1) then none of the messages on node 2 will get delivered since they won't be sucked over to node 1.

            I'd prefer being able to look at the distributed queue as a single virtual queue from which my consumer can take messages without having to worry from where a message originated.

            • 3. Re: Message redistribution
              timfox

               

              "martin.wickus" wrote:
              Precondition: JBM cluster consisting of two nodes posting messages to a clustered queue. The JBM cluster is formed across JBoss nodes.

              Step1. Start up a plain java process (configured with JBM client libraries and patched JBoss Remoting lib) which looks up a ClusteredConnectionFactory from the JBoss/JBM partition via JNDI, creates a connection then registers a MessageListener to receive messages. Let's say the resulting connection is attached to node 1 of the JBM cluster.

              Step2. Send a message from node 1 of the JBM cluster to the clustered queue. The message is immediately received by the message listener.

              Step3. Send a message from node 2 of the JBM cluster. The message is not received by the message listener.


              That should certainly work (it's the simplest possible redistribution scenario). Things to check:

              1) You have deployed your queue as clustered
              2) ClusterPullConnectionFactory is specified in the server peer config and is deployed
              3) Something is wrong in your JGroups config.

              Most likely is 3)

              Take a look at the JGroups site for some tips on how to work out whether the nodes are seeing each other etc.


              • 4. Re: Message redistribution
                timfox

                Also make sure you don't have other consumers on the nodes. Any local consumers will always get the message if they are not busy.

                • 5. Re: Message redistribution
                  timfox

                  Final thing to note:

                  In 95% of user deployments, message redistribution is NOT necessary.

                  Typically users deploy a homogeneous bank of MDBs (or other consumers) across the cluster - the same ones on each node. Client producing connections are also load balanced across the cluster. Since there's an even mix of producers/consumers across the cluster there's no need to distribute.

                  Distribution is uiseful in the case:

                  1) You have a bank of many consumers on a cluster but only, say., one or two consumers on a small number of nodes.

                  2) As above but swap producers with consumers

                  • 6. Re: Message redistribution
                    timfox

                    There is also a simple distributed queue example in the distro that we recommend you run (see install guide) - can you see if this works?

                    • 7. Re: Message redistribution
                      martin.wickus

                      1)You have deployed your queue as clustered
                      2) ClusterPullConnectionFactory is specified in the server peer config and is deployed
                      3) Something is wrong in your JGroups config.

                      1. - confirmed (via JMX attribute of destination in JMX-Console)
                      2. - confirmed (JMX mbean present in JMX-Console)
                      3. - Might need to investigate more here. However, I do find JBM clustering working for another scenario. I have a distributed topic and when I publish from any node, the resulting messages are consumed by all MDBs in cluster (1 such MDB per node).

                      Will check the distributed example and report back here.

                      • 8. Re: Message redistribution
                        martin.wickus

                        Disclaimer: I used my own queue (configured as per bottom of post)

                        I ran the test 5 times in a row:

                        1. Failure
                        2. Success
                        3. Success
                        4. Success
                        5. Failure

                        The two failures had the same error message:

                        C:\opt\downloads\jboss\jboss-messaging-1.4.0.SP3\jboss-messaging-1.4.0.SP3\examples\distributed-queue>ant -Djboss.home=C:\dev\abfo_home\DEV09\abf
                        o-1.2-clustering\abfo-jboss-1.2-clustering-dev-1 -Djboss.configuration0=abfo -Dexample.queue.name=OTL-LIFFEF-Send -Djava.naming.provider.url=jnp:
                        //10.142.204.77:59098ant -Djboss.home=C:\dev\abfo_home\DEV09\abfo-1.2-clustering\abfo-jboss-1.2-clustering-dev-1 -Djboss.configuration0=abfo -Dex
                        ample.queue.name=OTL-LIFFEF-Send
                        Buildfile: build.xml

                        identify:
                        [echo] ###########################################################################
                        [echo] # Running the DISTRIBUTED QUEUE example #
                        [echo] ###########################################################################
                        [echo] The queue: OTL-LIFFEF-Send
                        [echo] The client jar: ../..//jboss-messaging-client.jar

                        sanity-check:

                        init:

                        compile:

                        run:
                        [java] Distributed queue /queue/OTL-LIFFEF-Send exists
                        [java] The message was successfully sent to the distributed queue
                        [java] java.lang.NullPointerException
                        [java] at org.jboss.example.jms.distributedqueue.DistributedQueueExample.example(DistributedQueueExample.java:112)
                        [java] at org.jboss.example.jms.common.ExampleSupport.run(ExampleSupport.java:147)
                        [java] at org.jboss.example.jms.distributedqueue.DistributedQueueExample.main(DistributedQueueExample.java:167)
                        [java]
                        [java] #####################
                        [java] ### FAILURE! ###
                        [java] #####################


                        Queue configuration:

                        <!-- destinations for sending and receiving orders -->

                        <mbean code="org.jboss.jms.server.destination.QueueService"
                        name="jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send-DLQ"
                        xmbean-dd="xmdesc/Queue-xmbean.xml">
                        <depends optional-attribute-name="ServerPeer">
                        jboss.messaging:service=ServerPeer

                        jboss.messaging:service=PostOffice





                        true


                        <mbean code="org.jboss.jms.server.destination.QueueService"
                        name="jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send-ExpiryQueue"
                        xmbean-dd="xmdesc/Queue-xmbean.xml">
                        <depends optional-attribute-name="ServerPeer">
                        jboss.messaging:service=ServerPeer

                        jboss.messaging:service=PostOffice





                        true


                        <mbean code="org.jboss.jms.server.destination.QueueService"
                        name="jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send"
                        xmbean-dd="xmdesc/Queue-xmbean.xml">
                        <depends optional-attribute-name="ServerPeer">
                        jboss.messaging:service=ServerPeer

                        jboss.messaging:service=PostOffice

                        jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send-DLQ


                        jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send-ExpiryQueue






                        true

                        • 9. Re: Message redistribution
                          timfox

                          Right - this implies that distribution is not working for you - it works about half the time, and about half the time the consumer will be created (randomly) on the same node.

                          Almost certainly there's something wrong in your JGroups setup

                          • 10. Re: Message redistribution
                            timfox

                            BTW, can you please try with the preconfigured queues (i.e. do not change ANYTHING from the example) from the distro - i.e. don't use your own queue.

                            • 11. Re: Message redistribution
                              martin.wickus

                               

                              <mbean code="org.jboss.jms.server.destination.QueueService"
                               name="jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send"
                               xmbean-dd="xmdesc/Queue-xmbean.xml">
                               <depends optional-attribute-name="ServerPeer">
                               jboss.messaging:service=ServerPeer
                               </depends>
                               <depends>jboss.messaging:service=PostOffice</depends>
                               <attribute name="DLQ">
                               jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send-DLQ
                               </attribute>
                               <attribute name="ExpiryQueue">
                               jboss.messaging.destination:service=Queue,name=OTL-LIFFEF-Send-ExpiryQueue
                               </attribute>
                               <attribute name="SecurityConfig">
                               <security>
                               <role name="guest" read="true" write="true" />
                               </security>
                               </attribute>
                               <attribute name="Clustered">true</attribute>
                               </mbean>


                              • 12. Re: Message redistribution
                                martin.wickus

                                I added the test destinations to my configurations and it works with the default test queue. This would indicate something is wrong with the way my queue is configured. I'm about to make the configuration more similiar (remove DLQ, ExpiryQueue and SecurityConfig)

                                • 13. Re: Message redistribution
                                  martin.wickus

                                  I got it working, but something strange is afoot here.

                                  Although the JMX-Console showed my mbeans are clustered, the two entries in the table JBM_POSTOFFICE did NOT. In other words, the flag for the CLUSTERED column was set to 'N'.

                                  I deleted all entries from the JBM_POSTOFFICE table and restarted my nodes. This time, the clustered flags were set correct in the database and everything worked as expected. I ran the distributed-queue example with my own queue and it worked 10 times out of 10.

                                  I only recently clustered this destination, so I wonder if after making the config change, the entry didn't get updated in the database properly. I'll see if I can re-produce this and will report back.

                                  • 14. Re: Message redistribution
                                    martin.wickus

                                    I can confirm that updating a destinations' configuration (eg. setting the clustered attribute to true) doesn't seem to update the configuration in JBM_POSTOFFICE.

                                    1 2 Previous Next