-
1. Re: Paging, lastPageTransaction, and duplicate detection.
clebert.suconic Dec 31, 2008 1:51 PM (in response to timfox)The only reason for lastPageTransaction as you said was to avoid duplicated messages in case of a low-probable failure between the commit and the file removal.
If we can use that functionality to avoid duplications, it would be much better. -
2. Re: Paging, lastPageTransaction, and duplicate detection.
clebert.suconic Feb 9, 2009 6:14 PM (in response to timfox)One thing that I realized by accident today:
The duplicate detection is having a deep performance impact on paging.
With the LastPage stuff we would only add one record per-page, and delete it as soon as we had the confirmation of commit, having a minimal impact on the journal.
The duplicateID however, is storing one extra record perMessage. I know we will have the circularID and stuff, but that is just not as optimal wise as what we had before. -
3. Re: Paging, lastPageTransaction, and duplicate detection.
timfox Feb 10, 2009 3:55 AM (in response to timfox)It should be storing one per transaction, not per message
-
4. Re: Paging, lastPageTransaction, and duplicate detection.
clebert.suconic Feb 10, 2009 10:00 AM (in response to timfox)If you' re not using transactions on perfSender, it is storing one per message.
It would be better to do it per-depage. (or the transaction used on depaging).
Also... It would be even better (for reclaiming and compating), to delete the duplicate record as soon as the message is deleted. -
5. Re: Paging, lastPageTransaction, and duplicate detection.
timfox Feb 10, 2009 10:12 AM (in response to timfox)Yeah, we can do that as an optimisation - it should be fairly straightforward.
Deleting it however won't make any difference - they get deleted anyway when the cache gets full. -
6. Re: Paging, lastPageTransaction, and duplicate detection.
clebert.suconic Feb 10, 2009 11:45 AM (in response to timfox)Deleting it however won't make any difference - they get deleted anyway when the cache gets full.
It would have a difference on the timing the file could be reclaimed. We would use less files on the disk. (Possibly reusing more files from reclaiming).
Even after cleanup/compactor is implemented on the journal, the need of compacting or cleaning would be diminished. -
7. Re: Paging, lastPageTransaction, and duplicate detection.
timfox Feb 10, 2009 11:51 AM (in response to timfox)Duplicate detection is not only used in paging!