Transactional Summary & Holes on the journal

[clebert.suconic]

We talked through IRC today, about adding an update record when a file is reclaimed, in a way we would know how many records a transaction would have.

I just realized that this information is only available when loading.

We don't keep information about what transactions are current "alive" on the journal, besides when we are loading the journal.

We would need to add extra control for that, and we would need then to control reclaiming of transactions.

[clebert.suconic]

I'm making that post because when I was looking at the code, I was still considering the possiblity, but it wasn't possible.

I'm looking at a way to only store the number of Records at where the Commit/Prepare is being stored now.

[timfox]

A problem with that approach is you might have to update the commit record when you compact, since the number of tx operations in the file with the commit might have changed.

The commit record might be in the current file so that will be hard and undesirable.

If you just add update records you don't have to update the actual commit record.

And you have all the information you need to do this - when you compact the journal you know how many tx operations are in the file both before and after, so you can easily add an update record without having to store any extra information in memory.

[clebert.suconic]

"timfox" wrote:
A problem with that approach is you might have to update the commit record when you compact, since the number of tx operations in the file with the commit might have changed.

Yes.. I'm already considering that, but this shouldn't be a problem. Just a counter during compacting.

"timfox" wrote:


The commit record might be in the current file so that will be hard and undesirable.

Why that is a problem?

I'm not planning to compact the currentfile in any approach we choose

If you just add update records you don't have to update the actual commit record.

And you have all the information you need to do this - when you compact the journal you know how many tx operations are in the file both before and after, so you can easily add an update record without having to store any extra information in memory.

Actually I don't have any information for this at the moment.

The current implementation throws away the Transaction as soon as the commit/rollback arrives. I only know about the transaction while it is pendingin, during reload and now during compacting.


I would need to add a few collections to support that operation during reclaiming, and also make sure I remove records when the transaction is deleted (if I start making controls of the transaction).

When you consider that at the implementation level it starts getting more complicated.

[timfox]

What we need now is a detailed description of what you are proposing, it's still pretty vague to me.

I think we need this in order to move on

[clebert.suconic]

I'm writing the number of records at where the commit is located.

Example 1:

File1: 5 records
File2: 3 records
File3: 1 records
File4: 4 records and 1 commit


The commit will be:
COMMIT, 4

I won' t need to inform the other files on the summary any more, since I'm guaranteeing they are synced before writing the COMMIT. The only place I could have a hole now is at the current file.


Example 2:

The user took some time to commit, and there are no records at the top:


File1: 5 records
File2: 3 records
File3: 1 records
File4: 4 records
- User takes 2 hours to commit it:

File 10: Commit Record alone

The commit record on that case will be:

COMMIT, 0


what is a valid approach,

And compacting would work fine with this approach also.