7 Replies Latest reply on Aug 24, 2010 1:51 PM by lmcarriere

    HornetQ/JMS Large Scale Topology Design Questions

    lmcarriere

      Greetings, I have been investigating JMS, and in turn HornetQ, this past week as a solution for upgrading our existing product infrastructure.  I've already reviewed most of the user manual and examples, but some aspects are still unclear.

       

      Without going into too much detail, I have uploaded an abstract design here. http://www.tiikoni.com/tis/view/?id=f2be598

       

      Essentially we have thousands of clients and we manage and deploy our software from a central location within our corporate network.

       

      I'm proposing clustering HornetQ servers on our corporate side, while our clients have their own HornetQ server.  What concerns me is the configuration behind the queues, topics and core bridges for the internet connections.

       

      Here are some use-cases that concern me:

       

      1) Starting on the client side, application 1 pushes messages to the respective queues on the client hornetQ server.  It then bridges it to our corporate HornetQ cluster, which we consume and store in our enterprise database.

       

      - This seems straightforward, I believe the only way to achieve this is to specify the hosts in the cluster explicitly on each client hornetQ server (since it's over the internet we can't use any form of auto discovery), a minor concern is if we add more servers to the cluster, we need to upgrade all of our clients HornetQ configurations. One option of pushing those updates out would be via spacewalk. Am I missing any other alternative?

       

      2) Customers make changes to their software via the central administration web interface. Our application administration console produces an update message to our cluster.

       

      - These messages will usually be for a specific client, should these be posted to an application specific response queue? Will each client require it's own queue since there is no guarantee the intended client will be available (ie: network failure)? Or should we just be posting to a topic, even though only one client will need it?

      - How do the core bridge configurations work going back to each of our clients? Do we need to configure a remote-connector for each client and then a core bridge referencing each remote connector? I fear this would produce thousands of XML configurations and be unmanagable.

       

      3) A client starts up and requests its application code from our Calgary server.

       

      - It would post its initialize message to it's hornetQ server, which would relay it to the cluster and then our Calgary application consumer.

      - We would construct it's code in a large message and push it to the cluster, which would then transfer it to the client, does this impact the implementation of use-case #2?

       

      This turned into a longer message than I thought, sorry about that... any advice on if and how we should integrate JMS/HornetQ would be appreciated; or maybe I'm overcomplicating this whole thing.  Thanks!

       

      - Lindsey

        • 1. Re: HornetQ/JMS Large Scale Topology Design Questions
          clebert.suconic

          Can you ask more specific questions please?

           

           

          Putting your design, your database and everything you have there will make us do the consulting work. We try to help users at HornetQ usage, but it's impossible for us to design everybody's application.

           

          For example, instead of letting us to decide weather or not you should use specific IPs on your cluster.. you could ask us something more specific. (if not already documented of course). I'm kind of politely asking Read the manual first (I don't want to use the F word now ;-) ) although the F could mean fine ;-)

          • 2. Re: HornetQ/JMS Large Scale Topology Design Questions
            clebert.suconic

            1) So, a simpler question here would be: I have HornetQ clustered over the internet, how can I update the client's configurations?

             

            We are implementing update topology on the next version, without using UDP on the client. The servers are going to push the notifications.

             

            On the client, the topology is only used when creating new connections anyway. Maybe you should always get a connection from JNDI when instantiating new connections? (don't know how often you do it)

             

            for 2, and 3 are not specific HornetQ questions (allthough I would answer If I knew the answer)

            • 3. Re: HornetQ/JMS Large Scale Topology Design Questions
              lmcarriere

              Hi Clebert, thanks for the prompt reply.

               

              My apologies if it came accross like I'm expecting consultation from the community.  My intention was just to provide application specific context to the questions.  Regardless, the push notifications to update topology specifics sounds very promising and I'll be eagerly awaiting that. Questions 2 & 3 are touching more on general messaging standards but let me put it more in the context of HornetQ...

               

              If you have a cluster and there are thousands of remote nodes (connected via core bridges), and you need the cluster to relay messages to the nodes, does the cluster have to have specific configuration for the connections to each of those nodes like in the core bridge example?

               

              So in this case the hornetq-configuration.xml would look something like:

               

              - define connector to client1, client2, client<n>

              - define each applications queues/topics

              - define bridges to reference the client connector, one for each queue/topic

                  (I suppose the benefit here is each bridge could have a filter to redirect messages to the proper queue on the proper client)

               

              Is there any scalability concerns here? If we have 2000 clients, thats 2000 connectors and if there are 2 queues and a topic per application thats 6000 bridges per application configured in the cluster.

               

              In the meantime I'll be building my own prototypes off the HornetQ examples as that seems like the next logical step.

               

              - Lindsey

              • 4. Re: HornetQ/JMS Large Scale Topology Design Questions
                clebert.suconic

                With clustering, you define cluster connections...

                 

                 

                We use core-bridges between the clustering underneath.. but the instantiation goes by just defining the cluster connections:

                 

                 

                http://hornetq.sourceforge.net/docs/hornetq-2.1.2.Final/user-manual/en/html_single/index.html#clusters.cluster-connections

                 

                 

                 

                And until we deliver the push notification about the members, you can define the members of a cluster using this:

                 

                http://hornetq.sourceforge.net/docs/hornetq-2.1.2.Final/user-manual/en/html/clusters.html#d0e9918

                 

                 

                The connection factory is only used to spread the Connections among the members. make sure this is what you want.

                • 5. Re: HornetQ/JMS Large Scale Topology Design Questions
                  lmcarriere

                  Clebert, perhaps my question was still ambigious.  I am familliar with those references in the manual already, and my concern is not with the configuration of the cluster, but rather the configuration of the core bridge nodes.

                   

                  Let me pose it this way, with reference to the core bridge example... If I have one central node, and thousands of branch nodes, connected via the internet, to the central node.  Is the only way to configure them in the central hornetQ node to use extensive XML within the central node configuration?

                   

                  <configuration xmlns="urn:hornetq"
                    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                    xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">
                      <connectors>
                          <connector name="remote-connector-client_1">
                              <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
                              <param key="host" value="1.1.1.1"/>
                              <param key="port" value="5446"/>
                          </connector>
                          <connector name="remote-connector-client_2">
                              <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
                              <param key="host" value="2.2.2.2"/>
                              <param key="port" value="5446"/>
                          </connector>
                          <!-- and so on, all the way to -->
                          <connector name="remote-connector-client_100">
                              <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
                              <param key="host" value="100.100.100.100"/>
                              <param key="port" value="5446"/>
                          </connector>
                      </connectors>
                      <queues>
                          <queue name="jms.queue.central_to_clients">
                              <address>jms.queue.central_to_clients</address>
                          </queue>
                      </queues>
                      <bridges>
                          <bridge name="my-bridge-client_1">
                              <queue-name>jms.queue.central_to_clients</queue-name>
                              <filter string="client='client_1'"/>
                              <reconnect-attempts>-1</reconnect-attempts>
                              <connector-ref connector-name="my-bridge-client_1"/>
                          </bridge>
                          <bridge name="my-bridge-client_2">
                              <queue-name>jms.queue.central_to_clients</queue-name>
                              <filter string="client='client_2'"/>
                              <reconnect-attempts>-1</reconnect-attempts>
                              <connector-ref connector-name="my-bridge-client_2"/>
                          </bridge>
                          <!-- and so on, all the way to -->
                          <bridge name="my-bridge-client_100">
                              <queue-name>jms.queue.central_to_clients</queue-name>
                              <filter string="client='client_100'"/>
                              <reconnect-attempts>-1</reconnect-attempts>
                              <connector-ref connector-name="my-bridge-client_100"/>
                          </bridge>
                      </bridges>
                  </configuration>
                  


                  It just seems that when you get to thousands of clients with multiple queues/topics this would become unmanagable, though I'm not sure what the alternatives would be, or am I missing the point of core bridges altogether?

                   

                  Keep in mind, each distributed hornetQ server would already be bridged to the central node, their connector configuration might look like:

                   

                  <connectors>
                      <connector name="remote-connector-central">
                          <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
                          <param key="host" value="calgary.hq.company.com"/>
                          <param key="port" value="5446"/>
                      </connector>
                  </connectors>
                  <queues>
                      <queue name="jms.queue.clients_to_central">
                          <address>jms.queue.clients_to_central</address>
                      </queue>
                  </queues>
                  <bridges>
                      <bridge name="my-bridge-central_1">
                          <queue-name>jms.queue.clients_to_central</queue-name>
                          <reconnect-attempts>-1</reconnect-attempts>
                          <connector-ref connector-name="remote-connector-central"/>
                      </bridge>
                  </bridges>
                  


                  I guess where my uncertainty lies is with the bidirectional nature of messaging in a large, geographically distributed environment.  Am I understanding these concepts properly?  Thanks again for your advice.

                  • 6. Re: HornetQ/JMS Large Scale Topology Design Questions
                    clebert.suconic

                    I think I get your picture now. It sill a bit blur to me.. so, correct me if I still didn't get it.

                     

                    It seems you need some way to instantiate your bridges automatically...

                     

                    We have an interesting feature on the pipeline about peer mode : https://jira.jboss.org/browse/HORNETQ-87

                     

                     

                     

                    Maybe you could play with an embedded server, and instantiate the bridges through some code. Or you could also add your own component on hornetq-beans and instantiate the bridges yourself through java. That would at least work until we get the peer mode feature in place.

                     

                    Or even better.. Maybe you could contribute the feature... Depending if you get resources at your side.. and need the feature.. maybe you could speed up development.

                     

                    Or as I said.. .you could instantiate the bridges yourself.

                     

                     

                    Peer mode will be *probably* about sensing the peers and installing the bridges automatically (similarly to what we do on clustering).

                    • 7. Re: HornetQ/JMS Large Scale Topology Design Questions
                      lmcarriere

                      Thanks for the suggestions, and yes, it seems like you hit the nail on the head.

                       

                      I'll try and prototype all of these implementations and decide which, if any work best, or if we should contribute some form of peer mode to the project.  Regardless, I'll publish our findings.  You have been most helpful.

                       

                      - Lindsey