-
1. Re: Cluster Advice Required
timfox Jan 12, 2010 4:44 AM (in response to pieter.martin)So you're planning on using HornetQ to deliver the computations to your clients which will do the actual computations?
Are you intending on deploying the clients on the same machines as the HornetQ servers?
-
2. Re: Cluster Advice Required
pieter.martin Jan 12, 2010 6:07 AM (in response to timfox)I more thought of it as having one client that delivers computational jobs (message) to the cluster. The client would be on a different physical machine to the cluster. The nodes in the cluster then does the work.
There would then be a MDB that in its execute would do the calculation. Each calculation's result would either be put into a db or perhaps onto a disk shared by the cluster. Not quite sure yet of how we are going to store each calculations result.
After further discussion it seams the physical machine is a 64 core machine which they say sort of maps to 64 java threads which I am thinking maps to 64 nodes per machine in the cluster. There might be 2 or 3 of these machines so that would give us 64*2 / 3 nodes in the cluster.
However a node in the cluster might take up more than one thread so I am not sure this logic is solid.
Thanks
Pieter
-
3. Re: Cluster Advice Required
timfox Jan 12, 2010 6:16 AM (in response to pieter.martin)An MDB is a client too
So basically you want the processing to be done on the same nodes as the actual servers.
This should work, however you could also consider putting the HornetQ servers on separate machines and do the processing on other nodes.
This means the HornetQ machines won't be affected by the 100% CPU utilisation you're expecting for processing.
Also, any particular reason why you're considering MDBs?
I guess this is a matter of personal test but I'm not really a big JEE fan. I find MDBs can add complexity (and overhead) which you can avoid by just creating a simple JMS consumer.
-
4. Re: Cluster Advice Required
pieter.martin Jan 12, 2010 6:29 AM (in response to timfox)Ok yes I was planning on doing the calculation on the same machine. Did not even consider doing it on another machine. No reason for using a MDB. I used the wrong terminology there, we will be using a normal JMS consumer.
Ok thanks, here comes a hornetq super computer.
Pieter
-
5. Re: Cluster Advice Required
timfox Jan 12, 2010 6:33 AM (in response to pieter.martin)Sounds great
Tell me how it goes.
-
6. Re: Cluster Advice Required
timfox Jan 12, 2010 9:51 AM (in response to timfox)I think traditionally many compute grids or parallel cluster architectures (like the old Beowulf cluster), have used many nodes for the actual compute, but only have one (or two for redundancy) specialist nodes which deliver the compute instructions to the compute nodes.
Just delivering compute packets to other nodes is not a particularly resource intensive task, especially considering many computations take a significant time to complete, traffic might not be that high.
Consequently the machine housing the messaging server can often be isolated and less powerful than the actual compute nodes.
-
7. Re: Cluster Advice Required
pieter.martin Jan 18, 2010 7:06 AM (in response to timfox)Hi,
After some confusion on my side regarding hornetq and pooling I have decided to use jboss appserver and a compute MDB. This seems to me a simpler solution as the app server will pool the MDB making it unnecessary for me to run and manage many nodes/consumers.
As an aside the compute MDB will not be the only app deployed on jboss in which case the app server might be an overkill.
Thanks
Pieter
-
8. Re: Cluster Advice Required
timfox Jan 18, 2010 7:11 AM (in response to pieter.martin)I guess it's a matter of personal taste.
But I'd say having to manage a whole JBoss AS instance is a lot more to manage and a lot more overhead than creating a simple JMS client.
-
9. Re: Cluster Advice Required
pieter.martin Feb 11, 2010 7:47 AM (in response to timfox)Hi,
An update,
We are going it standalone now. Using Weld though to keep me happy with injection.
However it has been decided for better or worse that the compute grid is to compute anything now, not just the current problem.
So we entered into the world of dynamic classloading in order for the consumer to download the classes required to do whateva it wants to do.
Cheers
Pieter