11 Replies Latest reply on Sep 2, 2008 2:42 PM by clebert.suconic

    Thread usage in AIO layer

    timfox

      I'm looking at AsynchronousFileImpl and notice that it maintains a Thread per instance.

      This worries me, if we have a lot of files. Can't these be pooled somehow?

        • 1. Re: Thread usage in AIO layer
          clebert.suconic

          That's why we don't keep all the files opened. That's one thing I changed on the journal. We only keep the currentFile opened.

          If we have lots of files on the journal, you will awats only have two threads.
          \

          • 2. Re: Thread usage in AIO layer
            clebert.suconic

            We keep the currentFile, and a cache for a nextOpenFile.

            So, you aways have one file opened, and a cached one waiting when you need to open a new file. (So there is no wait on openFile inside the Journal).

            • 3. Re: Thread usage in AIO layer
              timfox

               

              "clebert.suconic@jboss.com" wrote:
              That's why we don't keep all the files opened. That's one thing I changed on the journal. We only keep the currentFile opened.

              If we have lots of files on the journal, you will awats only have two threads.
              \


              Can't completions come in for a previous file once we've already moved to the next file?

              Or do we block waiting for all completions when moving to the next file?

              If the latter, doesn't this introduce latency?

              • 4. Re: Thread usage in AIO layer
                clebert.suconic

                 

                "timfox" wrote:
                Can't completions come in for a previous file once we've already moved to the next file?


                Yes...

                When the journal needs a new file it will:

                - Immediately return the next file (cached). Writes will start on the next file without any waits.

                In an executor:
                A- It will wait the pending writes to finish before it can close the file, and then close the file (This won't affect any of the transactions. It's just that I need to wait executions to finish before I can close the file or I would loose data)
                B- It will place another file back to the nextOpenFile cache.


                I have tested this and I didn't see any latency because of that.



                BTW: (A) is being tested under one of the integration tests, as this used to cause a crash on the native layer before Alpha.

                • 5. Re: Thread usage in AIO layer
                  timfox

                  Isn't there a possibility that it moves to the next file, and it's still waiting for completions to come in on the last file, then it fills the next file too, then tries to move to the next one, but there is no cached file, because completions haven't all come in for the first file?

                  What happens in that case?

                  • 6. Re: Thread usage in AIO layer
                    clebert.suconic

                     

                    What happens in that case?


                    That could happen if you have a slow disk.

                    But if you have a slow disk, you would have waits on writing and waiting transactions to complete anyways. I don't think it is an issue based on the measures I've taken on the waitForAnOpenFile. I can take some meausres if you like.

                    There is on possible change: Start the native thread on the nextOpenFile only when the file was returned. We keep it opened but without a thread. That would save us one thread.

                    That means. AIO files would have three states. Opened, Active(poller thread running) and Closed.

                    • 7. Re: Thread usage in AIO layer
                      timfox

                      You didn't answer the question, "what happens in this case"? ;)

                      • 8. Re: Thread usage in AIO layer
                        clebert.suconic

                        If the file.close takes a lot of time to finish (more than what it would take to write a whole new file), the server will wait until a new file is opened.

                        But from what I have seen on my tests TimeCompletion < TimeWrite, i.e. by the time a new file is required you will already have a new file opened.
                        what means this condition shouldn't happen under regular circumstances.

                        That's why I think this is not an issue. It would only happen if the disk was really slow. (Maybe some other application sharing the disk with JBM, and that would cause delays anyway).

                        • 9. Re: Thread usage in AIO layer
                          timfox

                           

                          "clebert.suconic@jboss.com" wrote:
                          If the file.close takes a lot of time to finish (more than what it would take to write a whole new file), the server will wait until a new file is opened.


                          To clarify:

                          You're saying the move to next file would currently block until the old file could be closed?


                          But from what I have seen on my tests TimeCompletion < TimeWrite, i.e. by the time a new file is required you will already have a new file opened.
                          what means this condition shouldn't happen under regular circumstances.

                          That's why I think this is not an issue.


                          Yes it's unlikely, but it's possible and therefore is an issue.

                          Anything that might happen, however unlikely we must handle, rather than just barf.

                          • 10. Re: Thread usage in AIO layer
                            clebert.suconic

                             

                            To clarify:

                            You're saying the move to next file would currently block until the old file could be closed?


                            I aways have one file opened. There is no wait.

                            The only possibility waits on nextFile is if .close is taking more time than what it takes to write data to a file.

                            This would be a lack of resources problem from the hardware, and I can't do any magics. If the system/HD is slow because of overuse.. it is slow.. what can I do?

                            Yes it's unlikely, but it's possible and therefore is an issue.


                            What can I do if the hardware is not answering properly? There is nothing I can do here.

                            Opening several files would only worsen the situation on the hardware.

                            Anything that might happen, however unlikely we must handle, rather than just barf.


                            We won't barf... we would wait until the openedFileCache was restablished.

                            • 11. Re: Thread usage in AIO layer
                              clebert.suconic

                              A quick change on this...

                              It's not committed yet, as it will take me one or two days before I can commit my changes, but I'm changing this.


                              Now AIO will only start the NativePollerThread when the first write or read is done, and not when the file is opened.

                              This way we won't have any pollers running if there are no pending writes.

                              This will save only 1 thread. But starting the poller when the first write is done is a better behaviour IMO.