I can think of a couple alternatives (although, not sure how good they are).
1.) Put the MySql Master (or all of MySql boxes) in a Linux cluster. Imagine 2 boxes in a cluster. I'm pretty sure you can configure it so that one node is a "master", and if it dies, then the other node takes over the "Master's" IP address. You would probably want to separate your storage from any one server in the cluster. (i.e. using a SAN). Plus, I'm not sure if the server that "becomes the Master" can actually be used while it's in waiting (e.g., It may need to be dormant). However, with 3 boxes, you could have a 2-box Linux Cluster, with 1 backup master. If both masters go down, though, then the cluster would fail. Not sure if you can make all the boxes in a cluster "capable of becoming the master".
2.) I believe there are Linux-based Load Balancers with failover capabilities out there. So, you'd have "x" number of Jboss servers connecting to a single IP on the Linux Load balancer server, which would farm requests out to "y" number of MySql Master/Slave boxes NOT in a cluster (even Windows boxes). It seems like the Load-balancer would just heartbeat the downstream boxes, and if one doesn't answer, then mark it as down?
I would be curious to hear of any JBoss + Database specific technologies that might be able to address this, too. E.g., does JBoss Cluster technology have any ability to farm database requests out to multiple databse servers? I don't know the answer to that one.
Thanks for the reply sappenin. The latest topology for a MySQL cluster is a synchronous cluster where the mysqld SQL server is sitting on top of a clustered db engine which is spread across many machines. There is no master or backups. They are equal. This means each node can be used rather than just sitting and waiting for another to fail.
So ip takeover is not the answer since wouldn't allow usage of all db nodes.
Loadbalancer would work, yes, but getting too complex for a "simple" email instalation since requires 3 layers of hardware just to store an email.
What makes sense to me, as a simple solution, is that each JBMS server can use a particluar db node. But then there is the issue as posted of what happens to JBMS when it's db node goes down? There is still at least one more db running on another ip. JBMS just needs to start using a different one.
As you mention, if JBMS could manage a pool of connections to different databases in the db farm that would be nice.
MySQL has an HA option. It is not free though. A load balancer is not sufficient. Go to MySQL.com for more details.
Any idea how something like this would work with Postgresql?
Not sure what you are referring to by a non-free HA implementation. All I see at MySQL.com is info on the MySQL cluster architectur (like the white paper "A Guide to High Availability Clustering: How MySQL Supports 99.999% Availability" and of course the clustering section of the docs). Each node entry point (MySQL server) has it's own ip. No mention is made of MySQL servers taking over ips which would require a single MySQL server (the takeover server) to listen to multiple ips at once (not to mention that now one server has to handle twice it's load, even if 5 nodes in cluster). There is mention however of "applications" failing over to working MySQL nodes which seems like the most sensible solution. 4 nodes split the load of the downed 5th node.
Tell me, how can JBMS be such a great enterprise mail server if it can't failover to another datastore instance, which it relies on to store mail? There is a single point of failure at the JDBC connection layer, at least for that JBMS node.
And what happens when a JBMS node can't reach it's database? Does it stop listening for POP3 and SMTP connections (which would at least allow a load balancer in front of the mail server to direct traffic to another JBMS server) or does it just keep listening for connections and replying with errors (requiring multiple MX records so the sending mail server can use a different one)? Worse yet, it would keep spooling mail and throwing SQLExceptions becasuse the database is gone.
Any light you could shed on this issue, acoliver, would be appreciated... surely by more than just sappenin and I as I don't see any JBMS documentation on this issue.
Also, please correct me if I'm wrong anywhere above. Especially if it means there is not actually a free HA JBMS solution.
Yup you're right. JBMS does not provide reliability without reliable storage. There seems to be no free and reliable DBMS. You demand completely free (so Java isn't a good choice) and completely reliable (So mysql is not a good choice). So it seems that JBMS isn't right for you.
I haven't done any clustered JBAS stuff yet, but the EJBs in JBCS are able to be clustered using JBAS right? Just the data store is the issue?
What about C-JDBC? http://c-jdbc.objectweb.org/
I'm interested in HA for going forward, but haven't done any implementations using JBAS yet.
EJB clustering is not appropriate here. C-JDBC would be a lower guarantee of reliability and consistancy but I see no reason that you couldn't do it. Ultimately you need a reliable storage solution if you demand reliable storage.
Thanks for the clarification, Andrew!
It looks like Sequoia will not only do load balancing & failover (i.e, if one database cluster goes down, the others will pick up), but it also does database distribution/replication:
The database is distributed and replicated among several nodes and Sequoia balances the queries among these nodes
Assuming you don't need a HUGE amount of database storage (i.e., you can do with an amount of storage limited by a given server, say 3 500GB hard drives per box), then the storage here would seem to be pretty redundant. Imagine 3 database boxes, each with 1500GB of storage. The database would be replicated across the 3 database servers (using Sequoia), and if one database box goes down, Sequoia just doesn't redirect database requests to the dead box anymore.
Of course, if one were thinking of opening a webmail service where millions of users were granted 2GB of mail storage each, then such a solution would not scale. Instead, a SAN would probably be a better bet for storage redundancy/availability, and expansion.
Thanks for the tip osterday. Sequoia looks promising. This is the type of functionality necessary to allow JBMS to be an HA solution. In this case, it appears a MySQL cluster would not even be necessary as the updates are "multicast" to the db nodes at the Sequoia controler level. Surely there is a performance cost there, as pointed out by Andrew, but there will be such a cost for any HA solution. Don't get me wrong Andrew. I'm excited for JBMS and would love to use it but in my situation there is no use for a fast mail server that can't be redundant.
The other solution could be to have each JBMS server use a db node which is in a MySQL cluster. The MySQL cluster will keep data/mail in sync. Load balancer in front of JBMS nodes or MX round-robin, as long as JBMS fails to handle the mail requests when it's jdbc connection is down so client can choose another MX or lb can choose another server.
Thanks all for the input.
For clarification, the probelm is not specific to JBMS. You will have to come up with a reliable storage solution for any mail server. One configuration is to have JBMS use something locally for the queue persistence, but that has its own reliability issue. Ultimately you need to decide what your tollerance for failure is and pick a storage solution that meets those needs.