Paging, lastPageTransaction, and duplicate detection.

[timfox]

AFAICT paging currently uses lastPageTransaction recorded in the journal to prevent pages being processed more than once in case of failure.

This adds some complexity to paging.

JBM now contains automatic duplicate detection functionality which is enabled by just adding a property MessageImpl.HDR_DUPLICATE_DETECTION_ID to the message.

Any messages with the same value will be rejected.

I am thinking that we can leverage this functionality in paging, which will allow us to remove all the lastPageTransaction stuff and make the code simpler.

[clebert.suconic]

The only reason for lastPageTransaction as you said was to avoid duplicated messages in case of a low-probable failure between the commit and the file removal.

If we can use that functionality to avoid duplications, it would be much better.

[clebert.suconic]

One thing that I realized by accident today:

The duplicate detection is having a deep performance impact on paging.

With the LastPage stuff we would only add one record per-page, and delete it as soon as we had the confirmation of commit, having a minimal impact on the journal.

The duplicateID however, is storing one extra record perMessage. I know we will have the circularID and stuff, but that is just not as optimal wise as what we had before.

[timfox]

It should be storing one per transaction, not per message

[clebert.suconic]

If you' re not using transactions on perfSender, it is storing one per message.


It would be better to do it per-depage. (or the transaction used on depaging).

Also... It would be even better (for reclaiming and compating), to delete the duplicate record as soon as the message is deleted.

[timfox]

Yeah, we can do that as an optimisation - it should be fairly straightforward.

Deleting it however won't make any difference - they get deleted anyway when the cache gets full.

[clebert.suconic]

Deleting it however won't make any difference - they get deleted anyway when the cache gets full.

It would have a difference on the timing the file could be reclaimed. We would use less files on the disk. (Possibly reusing more files from reclaiming).

Even after cleanup/compactor is implemented on the journal, the need of compacting or cleaning would be diminished.

[timfox]

Duplicate detection is not only used in paging!