JournalCopier

[timfox]

I notice that when syncing the journal - you're copying the records one by one.

Isn't this going to be really slow?

Surely it would be faster just to copy the used files as actual files? I.e. just load them byte buffer by byte buffer and send them raw down the wire then save each chunk direct to disk?

[clebert.suconic]

The read method is the same we use on load. The read isn't going to be "very slow" I think.

I also thought about doing it this way because the backup won't need to load the journal at the end of the copy. The Journal will be ready to be used as soon as the last add is sent.

I'm also only sending the valid records. I'm not sending any deleted records... the same process as the compacting.


I could run a test to validate the performance if you want.

[timfox]

Are you sending each record across the wire in it's own packet?

[clebert.suconic]

Yes.. Allthough I could optimize it.

[timfox]

Standard ethernet MTU is 1500 bytes.

So if average record size is < 1500 bytes, then you're mostly sending empty bytes across the network.

This will hugely affect performance. Make sure you set tcp no delay to false to get around this.