Wildfly 10 too many "pipe" file descriptors

[luuzz]

Hello Guys,

We have just put in production an application deployed on Wildlfy 10 and we are very impressed by clustering features.

Everything is fine except that we have a lot of "pipe" file descriptors opened, above 200 on startup but quicly increase to 1500. It seems to increase per three files.

We increased the max opend files system parameter and actually don't have any crash for now  but we didn't get this behaviour on JBoss AS 7.

So we'd like to know if this is normal or if there is any leak.

When we make a netstat, everything seems fine so it doesnt seem to be a socket leak.

Any help please ?

Regards

[ctomc]

That is expected behavior.

on Linux each open socket connection, open file, etc count take open file descriptors.

When load on system increases, having more concurrent requests, it results in more file descriptors.

it also reduces when load decreases, but it first needs to get to some "baseline"

[luuzz]

Thanks for the quick answer.

I understand but what do you mean by "baseline" ?

Sorry my english is not that good.

Regards

[dmlloyd]

Using Undertow for the web server means we have a mix of blocking and non-blocking I/O; in order to simulate blocking I/O we generally use NIO selectors, cached to each thread.  On most UNIX-like platforms, each selector has a corresponding pipe (each of which uses two FDs) to manage its wakeup operation.  This means that you will normally have n*2 pipe descriptors open for every task thread which is doing blocking I/O.  It is appropriate to size up the maximum allowed number of file descriptors accordingly.

[luuzz]

Thank you for the explaination.

I am not sure the number of files decreases. Does it mean that there is a leak somewhere ?

Speaking of NIO selectors, while checking the threadump i saw some traces like :

default I/O-3" #97 prio=5 os_prio=0 tid=0x00007f88bc280000 nid=0xd283 runnable [0x00007f88b6dd1000]

[Server:host1i1]    java.lang.Thread.State: RUNNABLE

[Server:host1i1]     at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

[Server:host1i1]     at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

[Server:host1i1]     at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

[Server:host1i1]     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

[Server:host1i1]     - locked <0x00000000b14d5618> (a sun.nio.ch.Util$3)

[Server:host1i1]     - locked <0x00000000b14d5608> (a java.util.Collections$UnmodifiableSet)

[Server:host1i1]     - locked <0x00000000b14d54f0> (a sun.nio.ch.EPollSelectorImpl)

[Server:host1i1]     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

[Server:host1i1]     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

[Server:host1i1]     at org.xnio.nio.WorkerThread.run(WorkerThread.java:515)

Is this normal or should i be worried ?

Thanks in advance.

David Lloyd a écrit:

Using Undertow for the web server means we have a mix of blocking and non-blocking I/O; in order to simulate blocking I/O we generally use NIO selectors, cached to each thread. On most UNIX-like platforms, each selector has a corresponding pipe (each of which uses two FDs) to manage its wakeup operation. This means that you will normally have n*2 pipe descriptors open for every task thread which is doing blocking I/O. It is appropriate to size up the maximum allowed number of file descriptors accordingly.

David Lloyd a écrit:

Using Undertow for the web server means we have a mix of blocking and non-blocking I/O; in order to simulate blocking I/O we generally use NIO selectors, cached to each thread. On most UNIX-like platforms, each selector has a corresponding pipe (each of which uses two FDs) to manage its wakeup operation. This means that you will normally have n*2 pipe descriptors open for every task thread which is doing blocking I/O. It is appropriate to size up the maximum allowed number of file descriptors accordingly.

[dmlloyd]

The stack trace you posted is normal, it just means an I/O thread is waiting for work.

It is normal for the number of selectors to stay up.  Each thread caches its selector and keeps it alive for the life of the thread.

[luuzz]

Great!!

I need to read more about undertow.

I was afraid to have a leak somewhere in my code.

To summarize, we can say that

• if i set io-threads parameter to 10 and set max-task-thread to 900 that means that the system will handle up to 900 concurrent requests , each thread is able to handle many tasks (or requests)
• Selectors are used to stimulate blocking io

Am i correct ?

Tganks in advance

[dmlloyd]

Alassane Ba wrote:

Great!!

I need to read more about undertow.

I was afraid to have a leak somewhere in my code.

To summarize, we can say that

• if i set io-threads parameter to 10 and set max-task-thread to 900 that means that the system will handle up to 900 concurrent requests , each thread is able to handle many tasks (or requests)
• Selectors are used to stimulate blocking io

Am i correct ?

Tganks in advance

Almost correct.  If you have max-task-thread set to 900, that means the system will handle up to 900 concurrent blocking requests (such as servlet calls).  It can handle even more requests if the request is serviced with an Undertow non-blocking handler.

Otherwise yes, you are correct.

[luuzz]

Thank you again.

So the relation between io threads and task is clear.

One thread can handle many tasks.

One last thing i promise

During my tuning journey, came across the request-limit filter. When applied, does it use its own io thread pool or the default one configured in io subsystem ?

Regards