Setup of symmetric cluster in JBoss AS 6 no longer possible with HornetQ?

[sebastian.koske]

Hi, I recently evaluated a possible usage of HornetQ integrated in a JBoss 5.1 Cluster - mainly because I would like to use clustered grouping.I read the documentation and got the examples running. But still I'm still confused about the roadmap for HornetQ Integration with JBoss Clustering:

1.) Untill now we could run an JBoss Cluster on multicast with completely symmetric nodes (except port configuration an serverPeerId) and completely symmetric deployments. This feature is really essential to us. As I understand, this paradigm is completely broken since HQ-HA requires asymmetric Live-Backup Pairs. Is this really the final solution for JBoss AS 6 Clustering?

2.) Until now, it did not matter in wich sequence nodes are started or stopped. This seems also to be broken, since all backup nodes must be started before their corresponding Live Server. Again, will this still be the case in AS6 Final?

3.) In case of failover, I have to copy the journal to the live server, then restart the backup server and then restart the Live Server. Well, this is totally out of the question for production use in our case. Are their any alternatives?

4.) Regarding the server pair configuration, will there be any alternative to the static configuration via backup-connector-ref? I would rather use a discovery group.

5.)  Finally, I dont understand why it is necessary to configure a single node as local coordinator in a clustered JBoss Installation. Shouldn't this be an HA-Singleton with failover out-of-the-box?

Best regards, Sebastian Koske.

[clebert.suconic]

Cluster and Failover are distinct subjects with HornetQ.

Failover requires a backup server, but we're doing Active->Passive now.

We are improving live-backups considerabily at the next version.

4) That's already in place... you can just configure it statically.

5) That's for Message Grouping only I think, right? Is that what you're talking about?

[sebastian.koske]

Thanks, I can understand that one distingushes between failover, fault-tolerance and load-balancing - but clustering generally includes all of those requirements. So my impressions were correct: HornetQ will only support loadbalancing out-of-the-box within JBoss cluster, while JBoss messaging 1.4.x supported all of the features mentioned in a symmetric JBoss cluster.

This means that instead of one cluster one would have to configure two distinct clusters if fail-over and fault-tolerance are required (which they generally are). I must say this is not at all appealing. Are there any plans to re-join these two cluster approaches in the near future? This is a serious problem in my opinion.

5) yes it's about the single coordinator for clustered message grouping. Again, this breaks cluster symmetry. Is there any chance for this coordinator to be configured as an HA-Singleton?

[clebert.suconic]

You need to setup a central database with JBoss Messaging for Failover, so it's not out of the box with JBoss Messaging.

You can add a backup node on each JBoss Server, so you will have each application server with two processes, one backup and one live working on a ring of passive failover.

Active->Active failover is a big problem. you need several synchronization, it won't scale and bug prone. Having a two processes approach will give you what you want though.

5) I need to wait Andy Taylor to come from vacation early next week.. or I would need some time to research on this. (what would go beyond next monday anyway)

[sebastian.koske]

Yes of course you need a database, but that's already likely the case in production. So that's is not much of a requirement.

I understand that when you move from a remote shared storage (RDBS) to a local distributed one, that you have to think about efficient replication. So naturally you need backups.

But I dont't see that now you need two different flavors of processes with static configuration, that you now have to consider starting sequences of these processes and that you have to do all this manual work in case of failover. Also, if I would setup a ring, adding and removing nodes on the fly would not be possible. You would have to integrate those manually everytime. All this overhead just to get rid of the RDBS - that's not much of a good deal to me.

So why dont you just include a generic backup acceptor to every hornetq installation and add a backup detection facility via discovery to all live servers in a cluster? Of course this would require that a backup server could backup more than one life server in case of failover, but this should be possible I think - just make a distinct backup journal for every backed live server?

In fact I did experiment with a possible ring, but it raised another issue:

If the backup server is a Jboss itself, you can either strip the application deployment down until there are only the queue definitions left  (asymmetric deployment, not very practical). Or, if you have message driven beans in your application, reconfigure the JmsXA Connection Factory to connect remotely to your liveserver. This is also a lot of configuration overhead. Besides, I also had the problem that in JBoss 5.1 any backup configuration immediately switches to a liver server mode after startup, even if no MDBs are deployed. I believe that's because the JCA instantly connects to the Message System which is misinterpreted as failover.

So my point is this: I work in a company thats ships JBoss as part of your system to our customers and does the first level support. The customers are not at all experts and need the cluster setup to be as easy as possible (symmetric servers, symmetric deployments). They need to be able to start and stop any process at any time and they also need as few processes as possible, since all processes induce maintenance costs. With all this extra requirements we will not be able to sell our system! And that's problem since the anticipated featured like clustered grouping and last value queues could solve a lot of problems we had to find custom (hacked) solutions for until now. Our system is not vastly message driven and messgaging performance has never been a major problem. We could perfectly live with a DB-based solution if it saves us the overhead - but we are really interested in your new features.

[clebert.suconic]

RDBMS is slow for messaging...

Active->Active failover.. we know it doesn't work.. even with a database. there's tons of issues that we have seen.

All I can tell you is that we are improving Failover at the next versions... we will probably make it automatic on 2.3 and we will support it as part of EAP.

Even if we do a DB-based solution... we will still have active->passive failover. that's not going to change. (That's how most messaging system work anyway.. besides JBoss Messaging).

[sebastian.koske]

I was of course refering to the central database approach as in JBM 1.4 - where no replication is needed at all for failover and there is no need to distingish beteen active and backup servers. I get the impression that you are pretty much fixed with the idea that everyone will prefer the configuration-excessive distributed approach over the "slower" centralized. This is not the case! So migration to AS 6 is not possible then, unless we downgrade. Looking forward to any progress, though.

[clebert.suconic]

You can also use a shared disk over NAS. You won't need replication for failover.

We are providing improvements on failover on 2.2, and for 2.2. we will offer shared storage. and we will start working on replication for 2.3. (Or maybe still 2.2 but for early next year before it gets to GA).

[schnelzer]

Clebert, do you know of any documentation or threads discussing the details of setting up

You can add a backup node on each JBoss Server, so you will have each  application server with two processes, one backup and one live working  on a ring of passive failover.

We'd like to try setting this up.  I assume we'll need to define two HornetQ mbeans on each server (one active and the other a backup for an active on another server).  Here's a diagram of what we're thinking.  Thanks.

[ataylor]

currently thats not possible, however will be very soon. Its currently what im working hard on fixing, theres a few issues with naming, mbeans etc where the 2 servers clash. It should be working in trunk in the next day or 2 and in the next release.

[ataylor]

p.s. like the diagram, i may steal it

[schnelzer]

I could do what I want to (refer to diagram) if I ran the HornetQ servers outside of JBoss AS as standalone servers, right?  That may be the best way to go until you get it working cleanly in JBoss AS.  Thoughts?

Sure, you're welcome to the diagram.  I sent you a link to the Google diagram too.  You can edit it or make a copy for yourself.

[ataylor]

yes, having a dedicated AS instance for the backup will do the trick

[sebastian.koske]

Hi, I like the diagram. That's quite what I was expecting to work already, but i couldn't get a live server and a backup server running in the same process. Since the backup servers must be started before any live server - for whatever reason - there seemed to be a hen-egg-problem with this configuration. It would be great if any JBoss "all" configuration had a preconfigured live and backup server and the backup server would use autodetection by default to find their live servers. And of course, if a failed live server comes back, no prequisite restarts of other server should be needed. Then we could actually use HornetQ. Looking very much forward to that.

[schnelzer]

Sebastian, I definitely agree.  I would like to be able to define both a live node and a backup node (for another server) in a single hornetq-configuration.xml.  Defining the backup node would be optional.  If defined, 2 instances of HornetQ would be started (under the hood).  The fact that two instances were started would be somewhat transparent to someone deploying a HornetQ implementation.  This proposal assumes there is nothing in the internal design of HornetQ that prevents two instances running in the same JVM (so long as separate ports and journal-directories can be specified in the configuration file. 

There would have to be some extra management functionality added so that if a backup node is defined, it would watch to see when its master node became active and automatically start when it sees its master go active.  This would also imply that specifying the journal-directory for the backup node definition would be mandatory as it would not be the same as the master HQ node that is running the the same JBoss AS instance.

Sebastian, if I undertand your post correctly then we have the same design in mind.

I'd be interested to hear the thoughts from the development team regarding this proposal and if this is something you could use some help with.