OK, we've tested the system, and it was loaded with 2.5M scheduled jobs in just under 3 minutes.
That is a very big improvement from the 5 hours we started with.
We understand our use case is not the standard one for HornetQ. Our system has millions of jobs it needs to process a day repeatedly. The time they get processed is not "written in stone", so i can live with short delays. There are a few dousens machines that process them, so the quick distribution of the jobs and the ability to acknowledge or rolback a job is the thing we were looking for.
We actually looked at quartz, but working with a DB to store the persistent jobs made the work rather slow, at least once we had more than 1M jobs, and since we need to get the jobs in multiple VMs.
For now HornetQ, with the scheduled messages, seems to be the best solution for us, having a quick messaging system, that is also persistent and can also schedule future jobs.
In case i realize that having too many scheduled messages is too much for HornetQ (though i'm hoping that clustering can help with that), i'm thinking about combining the two together.
Having Quartz as a scheduler, and having one process working with it, but doing all the actual jobs distribution to the worker machines and using HornetQ, but i hope we won't have to get to that.
We wil update you all once we'll continue our tests.
We want to also note the great support we've been getting from the team at HornetQ, especially from Clebert Suconic, who was working hard to find out what the probelm was.
Levona & Ofer