Custom Pub/Sub Pattern, is HornetQ a good fit?

[dro_k]

Hi Everyone,

We need to build a bit customized version of the consumer/producer pattern where:

1- Producers generate records individually

2- The records should be durable and not lost

3- Consumers check the queue and take the items from the queue. But this is where things get a bit hairy:

a) Consumers should take 'n' items at a time and create "transactional" batches where either all the items get processed successfully or returned to the queue.

b) The batch size 'n' should change dynamically depending on the 'max-age' of the items in the queue. For example, if our default batch size if 10 but there are 5 items that have been waiting in the queue for more than the max-age the we should just create a smaller batch and process them.

We wanted to see if we HornetQ core is a good fit for this problem. If so, can you please point me to a good starting point?

Thanks,

Drew

[ataylor]

HornetQ will fit perfectly, although i dont think its really a custom patter, simply consume your messages in a transaction and either commit or rollback depending on your business logic outcome. you can use the message timestamp to decide whether to start processing a batch yet, this is set at the time the producer sends it

[dro_k]

Thanks Andy. In that case:

1- How can the consumers know how many messages are in the queue? Or they just have to keep taking and rolling back?

2- Would I need to worry about race conditions since we have many consumers and producers?

-- Drew

[clebert.suconic]

You shouldn't need to count messages other than management needs but there are management operations for that. There's no common API to manage any jams server.

You just receive a message asynchronously and that's it.

[dro_k]

Not sure I undestand what you mean. We are not using HornetQ with JMS, but HornetQ core directly.

[ataylor]

1- How can the consumers know how many messages are in the queue? Or they just have to keep taking and rolling back?

Is this part of your applications needs, if so there are management methods but tbh this will start making your code convoluted, I would try architecting your app so you dont need this info.

2- Would I need to worry about race conditions since we have many consumers and producers?

what race conditions are you talking about, again I think this is a question of how you write your client code.

Maybe if you give your use case we could help more.

[dro_k]

Hi Andy,

I'll try to explain my use case:

1. Users generate contents which need to be indexed using something like Solr or ElasticSearch

2. The issue with Solr or ElasticSearch is that their performance greatly drops if you try to index one item at a time. You get much better throughput if you batch index multiple documents.

3. We have a pretty strict requirements for Near Real Time search, meaning once a user creates the content it needs to be indexed and searchable in few minutes.

4. So the idea is to use HornetQ in between to queue these indexable contents and then have separate jobs that poll this queue and index the documents in batches.

5. But since the behaviour of the users is not consistent, we can't simply have a job that reads say 100 docs at a time and then populates the indexes, since at a slower traffic time, a document might stay in the queue for a long time if the batch size requirement is not met.

HTH,

Drew

[ataylor]

to be honest I think a database would be best for storing this then have an SQL select statement that pulls out what records you want to process, you could still use messaging for sending the records tho and maybe an MDB that puts them in a database. The consumer could use a request/reply pattern thru an MDB to retreive the batch of records to process.