8 Replies Latest reply on Oct 11, 2017 11:29 AM by Alassane Ba

    Wildfly 10 too many "pipe" file descriptors

    Alassane Ba Newbie

      Hello Guys,

      We have just put in production an application deployed on Wildlfy 10 and we are very impressed by clustering features.

      Everything is fine except that we have a lot of "pipe" file descriptors opened, above 200 on startup but quicly increase to 1500. It seems to increase per three files.

      We increased the max opend files system parameter and actually don't have any crash for now  but we didn't get this behaviour on JBoss AS 7.

      So we'd like to know if this is normal or if there is any leak.

      When we make a netstat, everything seems fine so it doesnt seem to be a socket leak.

      Any help please ?

      Regards

        • 1. Re: Wildfly 10 too many "pipe" file descriptors
          Tomaz Cerar Master

          That is expected behavior.

           

          on Linux each open socket connection, open file, etc count take open file descriptors.

          When load on system increases, having more concurrent requests, it results in more file descriptors.

           

          it also reduces when load decreases, but it first needs to get to some "baseline"

          • 2. Re: Wildfly 10 too many "pipe" file descriptors
            Alassane Ba Newbie

            Thanks for the quick answer.

            I understand but what do you mean by "baseline" ?

            Sorry my english is not that good.

            Regards

            • 3. Re: Wildfly 10 too many "pipe" file descriptors
              David Lloyd Master

              Using Undertow for the web server means we have a mix of blocking and non-blocking I/O; in order to simulate blocking I/O we generally use NIO selectors, cached to each thread.  On most UNIX-like platforms, each selector has a corresponding pipe (each of which uses two FDs) to manage its wakeup operation.  This means that you will normally have n*2 pipe descriptors open for every task thread which is doing blocking I/O.  It is appropriate to size up the maximum allowed number of file descriptors accordingly.

              • 4. Re: Wildfly 10 too many "pipe" file descriptors
                Alassane Ba Newbie

                Thank you for the explaination.

                I am not sure the number of files decreases. Does it mean that there is a leak somewhere ?

                 

                Speaking of NIO selectors, while checking the threadump i saw some traces like :

                 

                default I/O-3" #97 prio=5 os_prio=0 tid=0x00007f88bc280000 nid=0xd283 runnable [0x00007f88b6dd1000]

                [Server:host1i1]    java.lang.Thread.State: RUNNABLE

                [Server:host1i1]     at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

                [Server:host1i1]     at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

                [Server:host1i1]     at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

                [Server:host1i1]     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

                [Server:host1i1]     - locked <0x00000000b14d5618> (a sun.nio.ch.Util$3)

                [Server:host1i1]     - locked <0x00000000b14d5608> (a java.util.Collections$UnmodifiableSet)

                [Server:host1i1]     - locked <0x00000000b14d54f0> (a sun.nio.ch.EPollSelectorImpl)

                [Server:host1i1]     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

                [Server:host1i1]     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

                [Server:host1i1]     at org.xnio.nio.WorkerThread.run(WorkerThread.java:515)

                 

                Is this normal or should i be worried ?

                Thanks in advance.

                David Lloyd a écrit:

                 

                Using Undertow for the web server means we have a mix of blocking and non-blocking I/O; in order to simulate blocking I/O we generally use NIO selectors, cached to each thread. On most UNIX-like platforms, each selector has a corresponding pipe (each of which uses two FDs) to manage its wakeup operation. This means that you will normally have n*2 pipe descriptors open for every task thread which is doing blocking I/O. It is appropriate to size up the maximum allowed number of file descriptors accordingly.

                David Lloyd a écrit:

                 

                Using Undertow for the web server means we have a mix of blocking and non-blocking I/O; in order to simulate blocking I/O we generally use NIO selectors, cached to each thread. On most UNIX-like platforms, each selector has a corresponding pipe (each of which uses two FDs) to manage its wakeup operation. This means that you will normally have n*2 pipe descriptors open for every task thread which is doing blocking I/O. It is appropriate to size up the maximum allowed number of file descriptors accordingly.

                • 5. Re: Wildfly 10 too many "pipe" file descriptors
                  David Lloyd Master

                  The stack trace you posted is normal, it just means an I/O thread is waiting for work.

                   

                  It is normal for the number of selectors to stay up.  Each thread caches its selector and keeps it alive for the life of the thread.

                  • 6. Re: Wildfly 10 too many "pipe" file descriptors
                    Alassane Ba Newbie

                    Great!!

                    I need to read more about undertow.

                    I was afraid to have a leak somewhere in my code.

                    To summarize, we can say that

                    • if i set io-threads parameter to 10 and set max-task-thread to 900 that means that the system will handle up to 900 concurrent requests , each thread is able to handle many tasks (or requests)
                    • Selectors are used to stimulate blocking io

                    Am i correct ?

                    Tganks in advance

                    • 7. Re: Wildfly 10 too many "pipe" file descriptors
                      David Lloyd Master

                      Alassane Ba wrote:

                       

                      Great!!

                      I need to read more about undertow.

                      I was afraid to have a leak somewhere in my code.

                      To summarize, we can say that

                      • if i set io-threads parameter to 10 and set max-task-thread to 900 that means that the system will handle up to 900 concurrent requests , each thread is able to handle many tasks (or requests)
                      • Selectors are used to stimulate blocking io

                      Am i correct ?

                      Tganks in advance

                      Almost correct.  If you have max-task-thread set to 900, that means the system will handle up to 900 concurrent blocking requests (such as servlet calls).  It can handle even more requests if the request is serviced with an Undertow non-blocking handler.

                       

                      Otherwise yes, you are correct.

                      • 8. Re: Wildfly 10 too many "pipe" file descriptors
                        Alassane Ba Newbie

                        Thank you again.

                        So the relation between io threads and task is clear.

                        One thread can handle many tasks.

                        One last thing i promise

                        During my tuning journey, came across the request-limit filter. When applied, does it use its own io thread pool or the default one configured in io subsystem ?

                        Regards