1 Reply Latest reply on Jun 10, 2009 10:31 AM by clebert.suconic

    Journal pauses when creating new files

    timfox

      One thing I noticed when doing perf tests.

      If you set journal min files to a high value, then fine.

      However if it's not high enough and you send enough data then eventually it will run out of files and will create new ones.

      The file creation process seems much slower than it did before - did something change in this area?

      It gives noticeable pauses when creating new files.

        • 1. Re: Journal pauses when creating new files
          clebert.suconic

          Nothing changed on this regard.

          Writes are aways sent on executors.. (On AIO, the main thread never block on a write, unless it is waiting for a sync callback).

          There is aways a file opened, and another file is immediately created when that file is returned.


          Perhaps, the only thing that changed was... we can get better throughput on the disk. If your disk is maxed out (your laptop for instance) the file creation may be competing with the journal write process. If the disk throughput is maxed out, the time to create a new file may be higher than the ammount of time needed to fill up a file, on that case we will have eventual pauses waiting for new files to be opened.

          I aways have seen this happening before.. and I've done a lot of testing around this.


          But I will take a look, see if we can improve anything on the process of files-creation.