Message Replay?

[scameronr]

One of the things I'd like to be able to do with HornetQ is "replay" a set of durable messages sent to an address.  The main issue is that I need to be able to potentially replay multiple times, always starting from the first message sent to the address.  (first message during a day, everything gets restarted daily)

I can get close by creating a durable queue for the address that no one listens to, and then create a consumer and receive() on that queue to 'replay'.  The issue is that only works once, if I need to replay a second time I'm out of luck.

It seems like I could potentially implement a copy() function by listening to the queue and reposting, but I'd have to use a different address, and in general that seems to get hairy.

What I'd like to do is be able to create a new queue at any time, but have it start at the 'beginning' of the address messages.  I looked through the source, and it seemed like something similar goes on with journal loading, but there wasn't anything obvious.

Does anyone have any suggestions for implementing replay?

BTW, otherwise, I've been trying out the GA and things have been working great.  Thanks!

[clebert.suconic]

You could be using Browser, but I'm afraid you would be using a Messaging System like a RDBMS.

In your place I would rather persist data on the DB and recover from the DB instead of leaving it on a Messaging System's memory.

I'm not talking just for HornetQ. That would be an anti-pattern on any messaging system.

[timfox]

scameronr wrote:

One of the things I'd like to be able to do with HornetQ is "replay" a set of durable messages sent to an address.  The main issue is that I need to be able to potentially replay multiple times, always starting from the first message sent to the address.  (first message during a day, everything gets restarted daily)

I can get close by creating a durable queue for the address that no one listens to, and then create a consumer and receive() on that queue to 'replay'.  The issue is that only works once, if I need to replay a second time I'm out of luck.

It seems like I could potentially implement a copy() function by listening to the queue and reposting, but I'd have to use a different address, and in general that seems to get hairy.

What I'd like to do is be able to create a new queue at any time, but have it start at the 'beginning' of the address messages.  I looked through the source, and it seemed like something similar goes on with journal loading, but there wasn't anything obvious.

Does anyone have any suggestions for implementing replay?

BTW, otherwise, I've been trying out the GA and things have been working great.  Thanks!

If you consume messages in a tx, then rollback the tx, the messages will automatically get replayed - not sure if that is what you want though - perhaps you could describe your use case in more detail? I'm not sure I've really got it.

[scameronr]

Yes, I'm sorry - in retrospect I was pretty brief.  Here is a good explanation of what I'm looking for:

http://qpid.apache.org/queue-replay.html

As I'm already writing the messages to a HornetQ 'durable log' (using the libaio support- very sweet), it seems wasteful to turn around and write the messages from HornetQ to another 'durable log' (such as a relational DB, key/value store, etc) to support "replay" of those messages.  (E.g. By shoving them back into the HornetQ system and writing them *again* to a HornetQ durable log to be sent back to clients listening for replay).  If I have to do that to replay, ironically I'm considering using the existing HornetQ journal log implementation.  Since all I need for replay is a durable, append only log format for the messages.  (A Relational DB or even key-value store is overkill). 

The only other implementation I could find was HOWL: High Speed ObjectWeb Logger (http://howl.ow2.org/index.html), but it looks like it hasn't been updated in some time.

Thanks for the quick reply!

[scameronr]

As another follow up, it looks like AMQP supports this type of functionality via setting TTL on the messages, then if a listener is created after messages have been sent to the queue, it will still receive those messages, as long as the TTL is still valid.  (So if I set the TTL to a day, any new listener I created would get a "replay", all messages from the start).

See: http://issues.apache.org/jira/browse/QPID-1596

[adrian.brock]

I think the easiest way to implement this would be to have to a special kind of topic.

1) When you create the topic, it automatically creates a MASTER durable subscription.

2) When somebody creates a subscription to the topic (durable or otherwise) instead of getting an empty subscription

(which is the jms semantic) it is initialsed with all the messages that are still in the MASTER subscription.

3) Any other additional management features like being able to set a TTL for clearing the MASTER subscription.

[timfox]

I guess I still haven't got my head around this.

Doesn't just browsing (in JMS: queue browser) a queue accomplish the same goals?

Or even just consuming messages from a queue but not acknowledging them?

If they're not acknowledged they won't get removed, so next time you create a consumer, they will be "replayed".

Is there some subtle difference I am missing? Otherwise it seems to me this requirement is already met by the standard JMS API (and HQ core API)