MaxIO, MinFiles and JournalFileSize

[clebert.suconic]

When you initialize libaio, you also initialize the number of AIO positions you will have on the kernel. That is the maximum number of pending writes you could have on the kernel. There is an almost hard limit on the OS, I guess it's 25K (I say almost.. because the variable it's not documented, or if it is.. it's hard to find it).

After few tests I have done, I've been able to get very good numbers, using the StoreManager directly. But the best number I could get to my disk was 10000. With that we can't have more than 2 files opened with AIO. Since the journal is append only, I guess it would be okay to make it a little bigger, since we won't make any reads on the file.


I will make the MaxIO configurable on the Configuration, but IMO the default for min files should be 2, and the size of each journal at 100M.

Putting the file at 100M would minimize the number of closes/opens you would do in production also.

When you close a file in AIO, you need to wait all the pending writes to complete before you can actually close it and move to the next file.

(Maybe I should use an executor to close the file to avoid the eventual latency, but if you could minimize the number of closes on AIO it would still a good practice IMO)

[ataylor]

whatever the best configuration turns out to be, we need document this well and have multiple examples for different scenarios that a user can easily switch between without having to fully understand the ins and outs AIO. i.e. best config for heavy throughput, low throughput with large messages etc etc.

[clebert.suconic]

I really want to keep freeFiles closed.

With that we can have as many free files as we want without having the side effect of this formula:

NumberOfOpenedFiles * MAX_AIO_PER_FILE <= GLOBAL_MAX_AIO



As for now, I have made MAX_AIO = 2000, so our testsuite can complete without any problems.

[ataylor]

I really want to keep freeFiles closed.

Is there something stopping us from doing this?

[clebert.suconic]

"ataylor" wrote:

I really want to keep freeFiles closed.

Is there something stopping us from doing this?

Yes.. Tim Fox! :-D

[timfox]

The idea with freeFiles being pre-opened is to prevent a pause when the journal moves to the next file as it opens the file.

[clebert.suconic]

"timfox" wrote:
The idea with freeFiles being pre-opened is to prevent a pause when the journal moves to the next file as it opens the file.

We could have only the first freeFile opened.

A checkout from freeFile would aways return instantly while another thread would open the next first file. Same way as we would have another thread closing the file. (since that represents a big pause on AIO).

And I would make the files bigger by default also.. as I said before... to avoid opening and closing them so often under high load.

[ataylor]

or we could have 2 files open, one being used and one ready. The thread that closes file 1, opens file 3 ready and so on. This means that we always have MAX_AIO_PER_FILE = GLOBAL_MAX_AIO/2

[timfox]

"ataylor" wrote:
or we could have 2 files open, one being used and one ready. The thread that closes file 1, opens file 3 ready and so on. This means that we always have MAX_AIO_PER_FILE = GLOBAL_MAX_AIO/2

Sounds like a good idea

[clebert.suconic]

Yes... that's what I meant when I said to keep the first freeFile opened.



The freeFile concept will be used for the pre allocations only, and we will aways have 2 files opened. (The current and next)