5 Replies Latest reply on Aug 30, 2015 10:46 PM by jbertram

    Is message grouping supported within a symmetric cluster

    hardy.ferentschik

      Hi,

       

      I am running Wildfly 9.0.0.Final (with HornetQ 2.4.7.Final) using a symmetrical cluster configuration using two Wildfly instances, each running two HornetQ servers (one live, and one being the backup for the other's node live server). The config I am using comes from here - https://github.com/foogaro/wildfly-cookbook/tree/master/cluster-jms-replication (example code from WildFly Cookbook).

       

      On top of this configuration I want to add message grouping. I am setting the JMSXGroupID in the JMS message and also added the following pieces

      of XML:

       

      <grouping-handler name="my-grouping-handler">
          <type>LOCAL</type>
          <address>jms</address>
          <timeout>5000</timeout>
      </grouping-handler>
      

       

       

      <grouping-handler name="my-grouping-handler">
          <type>REMOTE</type>
          <address>jms</address>
          <timeout>5000</timeout>
      </grouping-handler>
      

       

      The LOCAL type I specify on the live node of the first Wildfly instance and its backup node on the second. The other two HornetQ servers use REMOTE.

       

      When I start the servers and send some test messages I can see that message grouping works as expected. However, when I then stop the first server and try to send test messages again, I can look up the queue, but the message sending hangs and will eventually time out on the client side with an exception. In the server log I can see

       

      20:22:58,346 WARN  [org.hornetq.core.server] (default I/O-5) HQ222161: Group Handler timed-out waiting for sendCondition
      

       

      I am wondering whether the setup I am trying to configure is supported and whether it is the right approach regarding its grouping-handler configuration. Should the backup node on server two also take over as LOCAL grouping-handler on a fail-over or not?

        • 1. Re: Is message grouping supported within a symmetric cluster
          jbertram

          The setup sounds correct to me.  Are you sure the backup with the LOCAL handler is actually becoming active when the live with the LOCAL handler is stopped?

          • 2. Re: Is message grouping supported within a symmetric cluster
            hardy.ferentschik

            The setup sounds correct to me.

             

            Cool. Just wanted to make sure that it is supported.

              Are you sure the backup with the LOCAL handler is actually becoming active when the live with the LOCAL handler is stopped?

             

            Actually I think it did not. This might be related to the general problems I had with my cluster setup. Basically fail-over would not work reliably (see also [WFLY-4929] Hornetq failover doesn't work - JBoss Issue Tracker). After many other experiments I switched from http acceptors/connectors to netty. This seems to make a difference and the setup works (still needs more testing). I am not sure whether the settings I am using for the http connectors/acceptors are bad or whether this type of config is generally less reliable than the netty approach.

            • 3. Re: Is message grouping supported within a symmetric cluster
              jbertram

              I would be surprised if there was much difference between the HTTP connector and a plain Netty connector.  The HTTP connector is basically just a thin wrapper around a Netty connector to allow the server to only expose the HTTP port (i.e. single port for all services).  That said, problems occur in surprising places all the time.  If you have a test-case to reliably reproduce the unexpected result then I recommend you attach it either here or to the appropriate JIRA.

               

              Keep in mind that a graceful server shutdown will not trigger a fail-over unless you have <failover-on-shutdown>true</failover-on-shutdown>.  See the documentation for more details on this.

               

              Also, depending on how much message grouping you are doing and large the groups are it may not make sense to even have a cluster.  For example, if you are using a small number of groups and most of the messages are in one of those groups then that means most of your message consumption is going to be serial.  Serial consumption is a significant performance bottleneck (although obviously required in some use-cases).  A cluster is meant to increase message throughput and decrease latency, but if you have mostly serial consumption then you're actual performance needs would likely be served by a single server.

               

              Even if you don't have mostly serial consumption your performance needs still might be served by a single server.  A single HornetQ server can handle millions of messages per second.  Users often jump to the conclusion that clusters are necessary for their performance needs before actually doing any goal-setting or benchmarking, but this can lead to a lot of wasted resources.  Don't let that happen to you.

              1 of 1 people found this helpful
              • 4. Re: Is message grouping supported within a symmetric cluster
                hardy.ferentschik

                Thanks for the great reply. This helped a lot and clarified a few other questions I had.

                 

                I would be surprised if there was much difference between the HTTP connector and a plain Netty connector.  The HTTP connector is basically just a thin wrapper around a Netty connector to allow the server to only expose the HTTP port

                 

                Interesting. I will try whether I can re-produce this and create some sort of test case. I just know that I had problems testing the setup, but once I switched to netty in the acceptors/connectors it worked. I am pretty sure that I have not changed anything else in the config. Maybe some of the settings in the HTTP connectors/acceptors were causing the problem. It is really all a bit hard to put together and the examples one finds on the internet are all a bit different. In my case I started off with an example out of the WildFly Cookbook.

                 

                Keep in mind that a graceful server shutdown will not trigger a fail-over unless you have <failover-on-shutdown>true</failover-on-shutdown>

                 

                Sure, I have this option set.

                 

                Also, depending on how much message grouping you are doing and large the groups are it may not make sense to even have a cluster.  For example, if you are using a small number of groups and most of the messages are in one of those groups then that means most of your message consumption is going to be serial.  Serial consumption is a significant performance bottleneck (although obviously required in some use-cases).  A cluster is meant to increase message throughput and decrease latency, but if you have mostly serial consumption then you're actual performance needs would likely be served by a single server.

                 

                Thanks, this actually answers one of my own concerns and question I wanted to follow up on. In fact I also believe that in my case a symmetric cluster with multiple active nodes is not required. I Would like to have, however, a single server with a standby backup. Unfortunately, this configuration seems at the moment not possible due to [HORNETQ-1489] HQ222116: unable to start broadcast group - JBoss Issue Tracker. I cannot just configure a single server in the WildFly configuration. I think this is fixed in Artemis and will eventually make it into Wildfly 10, but for Wildfly 9 I am not sure how to set a simple active server + back up.

                 

                Even if you don't have mostly serial consumption your performance needs still might be served by a single server.  A single HornetQ server can handle millions of messages per second.  Users often jump to the conclusion that clusters are necessary for their performance needs before actually doing any goal-setting or benchmarking, but this can lead to a lot of wasted resources.  Don't let that happen to you.

                 

                It won't ;-) Thanks for these valuable tips.

                • 5. Re: Is message grouping supported within a symmetric cluster
                  jbertram

                  I Would like to have, however, a single server with a standby backup. Unfortunately, this configuration seems at the moment not possible due to [HORNETQ-1489] HQ222116: unable to start broadcast group - JBoss Issue Tracker. I cannot just configure a single server in the WildFly configuration. I think this is fixed in Artemis and will eventually make it into Wildfly 10, but for Wildfly 9 I am not sure how to set a simple active server + back up.

                  If you're having trouble with JGroups then (as I said before) don't use JGroups.  You can use the built-in UDP clustering available in HornetQ or configure static connectors.  The documentation covers both of these options.