HORNETQ-316 Make cluster discovery pluggable

[igarashitm]

https://issues.jboss.org/browse/HORNETQ-316

http://community.jboss.org/thread/148457

To cluster discovery get work on amazon AWS, we need 2 steps:

1. make cluster discovery pluggable and add JGroups discovery plugin
2. verify at least the one of GossipRouter, FILE_PING and S3_PING woks fine on amazon AWS

First I'm considering about step-1. Please review my idea.

• classes to add

• org.hornetq.core.cluster.impl.DiscoveryGroupFactory
  • : if config has valid "class" attribute, create it. otherwise, create existing DiscoveryGroupImpl
• org.hornetq.core.server.cluster.impl.BroadcastGroupFactory
  • : if config has valid "class" attribute for broadcast-group, create it. otherwise, create existing BroadcastGroupImpl
• org.hornetq.integration.discovery.jgroups.JGroupsDiscoveryGroupImpl
• org.hornetq.integration.discovery.jgroups.JGroupsBroadcastGroupImpl

• behavior changes on existing classes
  • org.hornetq.core.client.impl.ServerLocatorImpl#initialize() call DiscoveryGroupFactory#createDiscoveryGroup() instaed of call new DiscoveryGroupImpl() directly.
  • org.hornetq.core.server.cluster.impl.ClusterManagerImpl#deployBroadcastGroup() call BroadcastGroupFactory#createBroadcastGroup() instead of call new BroadcastGroupImpl() directly.

• hornetq-configuration.xml changes

  •    <broadcast-groups>

          <broadcast-group name="my-broadcast-group">

             <param key="class" value="org.hornetq.integration.discovery.jgroups.JGroupsBroadcastGroupImpl"/>

             <param key="jgroups-channel-name" value="hornetq-discovery"/>

             <!-- attributes for BroadcastGroupImpl

             <param key="group-address" value="231.7.7.7"/>

             <param key="group-port" value="9876"/>

             <param key="broadcast-period" value="100"/>

             <param key="connector-ref" value="netty-connector"/>

             -->

         </broadcast-group>

       </broadcast-groups>

       <discovery-groups>

          <discovery-group name="my-discovery-group">

             <param key="class" value="org.hornetq.integration.discovery.jgroups.JGroupsDiscoveryGroupImpl"/>

             <param key="jgroups-channel-name" value="hornetq-discovery"/>

             <!-- attributes for DiscoveryGroupImpl

             <param key="group-address" value="231.7.7.7"/>

             <param key="group-port" value="9876"/>

             <param key="refresh-timeout" value="10000"/>

             -->

          </discovery-group>

       </discovery-groups>

• TODOs
  • abstraction of org.hornetq.api.core.DiscoveryGroupConfiguration
    • currently, tightly coupled with existing DiscoveryGroupImpl
  • consider where to put the jgroups stack configuration for client, and standalone server
    • in JBossAS, we can put it in deploy/cluster/jgroups-channelfactory.sar/META-INF/jgrouls-channelfactory-stacks.xml

any advice would be appreciated

Thanks

[keteracel]

here's a patch which uses JGroups for discovery rather than custom multicast. (http://paste.pocoo.org/show/318348/)

Two observations:

1) I think that as JGroups deals with a Channel (rather than discovery and broadcast), DiscoveryGroup and BroadcastGroup should be merged, with an option to turn broadcast off for clients.

2) This could be completely pluggable so people can use whatever they like.

So this patch just replaces the original multicast with jgroups which is enough for me to continue, allows multicast for people who want it, but ties people to jgroups (not too bad an idea).

Thanks,

Paul.

[ataylor]

theres also static discovery that should be factored in, currently this is separate code in server locator (see StaticConnector), this basically takes a list of servers and tries them all at the same time and waits for one to connect.

[clebert.suconic]

People can use JGropus as a plugin. We don't want the dependency on hornetq core.

[clebert.suconic]

What you're suggesting is that StaticConnector should also be a plugin point, right?

It makes sense.

[ataylor]

Clebert Suconic wrote:

What you're suggesting is that StaticConnector should also be a plugin point, right?

It makes sense.

correct

[igarashitm]

OK, update to deal with StaticConnector and config abstraction:

• classes to add
  • org.hornetq.core.cluster.DiscoveryGroupConfiguration
    • has method : String getClassName()
  • org.hornetq.core.cluster.impl.StaticDiscoveryGroupConfigurationImpl extends DiscoveryGroupConfiguration
  • org.hornetq.core.cluster.impl.SimpleUDPDiscoveryGroupConfigurationImpl extends DiscoveryGroupConfiguration

• org.hornetq.core.cluster.impl.DiscoveryGroupFactory
  • has method : DiscoveryGroup createDiscoveryGroup(DiscoveryGroupConfiguration)
    • return Class.forName(DiscoveryGroupConfiguration#getClassName()).newInstance()
• org.hornetq.core.server.cluster.BroadcastGroupConfiguration
  • has method : String getClassName()
• org.hornetq.core.server.cluster.impl.SimpleUDPBroadcastGroupConfigurationImpl extends BroadcastGroupConfiguration
• org.hornetq.core.server.cluster.impl.BroadcastGroupFactory
  • has method : BroadcastGroup createBroadcastGroup(BroadcastGroupConfiguration)
    • return Class.forName(BroadcastGroupConfiguration#getClassName()).newInstance()
• org.hornetq.integration.discovery.jgroups.JGroupsDiscoveryGroupConfigurationImpl extends DiscoveryGroupConfiguration
• org.hornetq.integration.discovery.jgroups.JGroupsBroadcastGroupConfigurationImpl extends BroadcastGroupConfiguration
• org.hornetq.integration.discovery.jgroups.JGroupsDiscoveryGroupImpl implements DiscoveryGroup
• org.hornetq.integration.discovery.jgroups.JGroupsBroadcastGroupImpl implements BroadcastGroup

• behavior changes on existing classes
  • ServerLocator call DiscoveryGroupFactory#createDiscoveryGroup() instaed of call new DiscoveryGroupImpl() directly
  • ClusterManagerImpl call BroadcastGroupFactory#createBroadcastGroup() instead of call new BroadcastGroupImpl() directly
  • add DiscoveryGroup#selectConnector()
  • remove HornetQClient#createServerLocator{With,Without}HA(TransportConfiguration...)
    • use HornetQClient#createServerLocator{With,Without}HA(DiscoveryGroupConfiguration) instead. pass the StaticDiscoveryGroupConfiguration as param.

• hornetq-configuration.xml changes
  • same as I posted first

I think that solve with above... anything else I should consider ?

Thanks.

[keteracel]

That sounds great to me!

Any ETA on when this work will be done? If you would like contributions, I'm happy to help.

Thanks,

Paul.

[igarashitm]

I'm now working on http://anonsvn.jboss.org/repos/hornetq/branches/HORNETQ-316/

I've just finished to write a prototype, and I'll test & debug from today. any comment would be appreciated.

Thanks.

[keteracel]

Awsome.

My only question is regarding how this works with JNDI. It looks like it'll require the jgroups config file to be in the same location on subsequent hosts that use JNDI for configuration. When I created my 'hacked' version I pushed the jgroups config file contents into JNDI so you could remove the config dependency. Not sure if that was the right thing, but thought I'd hit you up anyways.

But yes, this is looking pretty spot on what I'd need!

Thanks,

Paul.

[keteracel]

I'm also wondering if it's worth collapsing Broadcast and Discovery into a single Class with an option to join the cluster (broadcast) or just discover the cluster. It seems that it would be more optimal to use the JChannel as a channel rather than having two of them - one for broadcast and one for discovery.

That may be a future refactoring though

[bbansal]

Folks,

Is it pushed to trunk yet ? I am running hornetq-2.2.5 can it be back ported to it ?

[keteracel]

When is this targetted for release?

We didn't go with HornetQ for my last project because of this issue. I'd really like to use HornetQ on my current project but again we can't rely on multicast for discovery and a static list = downtime.

I really appreciate the work you are doing to add this to HornetQ and am eager to start using it!

Paul.

[bbansal]

Well,

We at Groupon are going in production with a simple design which solve this problem.

Producer --> Load-balancers ---> Cluster of HornetQ servers 

The producer client refreshes connection every 5 mins internally and hence any new machine added or deleted is handled easily.

Consumer  ------>

                  ------>      All hornetQ servers and consume from all

Consumer connect to all HornetQ servers in parallel and consume from all, The consumer client reads a list of active servers and refresh connection every 5 mins etc.

Each hornet-q machine in cluster is more or less unaware of others and performs wonderfullly. We have clients in java/ruby let me know if there is interest in open sourcing these out.

[keteracel]

That is a valid workaround. However, one of the reasons we want a HornetQ cluster is for the clustering and loadbalancing without the need for another piece of infrastructure / more network hops.