Dear all,

We have a very complex problem using FUSE ESB 6.3 redhat-630254. For some reasons, on each cluster's node, the number of file descriptor used exceed the limit (more than 65000).

Our architecture is based on 5 nodes with the bundle service deployed on each one (using Fabric profile).

We are using :

camel vesion 2.17.0.redhat-630187

camel-netty4-http version 2.17.0.redhat-630187

Below the exception :

2017-10-16 14:09:02,577 | ERROR | entExecutorGroup | DefaultErrorHandler              | 378 - org.apache.camel.camel-core - 2.17.0.redhat-630254 | Failed delivery for (MessageId: ID-SVR-ESB-FUSE-SLAVE-3-49950-1508016636199-4-2511736 on ExchangeId: ID-SVR-ESB-FUSE-SLAVE-3-49950-1508016636199-4-2511737). Exhausted after delivery attempt: 1 caught: io.netty.channel.ChannelException: Unable to create Channel from class class io.netty.channel.socket.nio.NioSocketChannel. Processed by failure processor: FatalFallbackErrorHandler[Pipeline[[Channel[SetHeader(CamelHttpResponseCode, 500)], Channel[SetHeader(Content-Type, application/json)], Channel[DelegateSync[com.ldlc.commons.technical.esb.route.CommonsExceptionRoute$$Lambda$18/279035000@639d36d6]], Channel[Marshal[org.apache.camel.component.jackson.JacksonDataFormat@3f4fd890]]]]]n  | n  | Message Historyn  | ---------------------------------------------------------------------------------------------------------------------------------------n  | RouteId              ProcessorId          Processor                                                                        Elapsed (ms)n  | [route78           ] [route78           ] [http://0.0.0.0:10009/api/rest/Configuration/user/%7BsiteId%7D/%7BuserId%7D?htt] [        31]n  | [route78           ] [restBinding64     ] [                                                                              ] [         0]n  | [route78           ] [route78           ] [direct:configurationService.getByUserId                                       ] [        31]n  | [route95           ] [removeHeader75    ] [removeHeader[CamelHttpPath]                                                   ] [         0]n  | [route95           ] [configurationBackE] [                                                                              ] [        30]n  | [                  ] [setHeader327      ] [setHeader[CamelHttpResponseCode]                                              ] [         0]n  | [                  ] [setHeader328      ] [setHeader[Content-Type]                                                       ] [         0]n  | [                  ] [process133        ] [Processor@0x639d36d6                                                          ] [        27]n  | [                  ] [marshal140        ] [marshal[org.apache.camel.model.dataformat.JsonDataFormat@76c98c8b]            ] [         2]n  | n  | Stacktracen  | ---------------------------------------------------------------------------------------------------------------------------------------

io.netty.channel.ChannelException: Unable to create Channel from class class io.netty.channel.socket.nio.NioSocketChannel

        at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:457)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:308)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.bootstrap.Bootstrap.doConnect(Bootstrap.java:135)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:117)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at org.apache.camel.component.netty4.NettyProducer.openConnection(NettyProducer.java:420)[395:org.apache.camel.camel-netty4:2.17.0.redhat-630254]

        at org.apache.camel.component.netty4.NettyProducer$NettyProducerPoolableObjectFactory.makeObject(NettyProducer.java:550)[395:org.apache.camel.camel-netty4:2.17.0.redhat-630254]

        at org.apache.camel.component.netty4.NettyProducer$NettyProducerPoolableObjectFactory.makeObject(NettyProducer.java:546)[395:org.apache.camel.camel-netty4:2.17.0.redhat-630254]

        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)[412:org.apache.commons.pool:1.6.0.redhat-9]

        at org.apache.camel.component.netty4.NettyProducer.process(NettyProducer.java:206)[395:org.apache.camel.camel-netty4:2.17.0.redhat-630254]

        at org.apache.camel.component.netty4.http.NettyHttpProducer.process(NettyHttpProducer.java:52)[396:org.apache.camel.camel-netty4-http:2.17.0.redhat-630254]

        at org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:124)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:341)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:119)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]

        at org.apache.camel.component.netty4.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:138)[395:org.apache.camel.camel-netty4:2.17.0.redhat-630254]

        at org.apache.camel.component.netty4.handlers.ServerChannelHandler.channelRead0(ServerChannelHandler.java:109)[395:org.apache.camel.camel-netty4:2.17.0.redhat-630254]

        at org.apache.camel.component.netty4.http.handlers.HttpServerChannelHandler.channelRead0(HttpServerChannelHandler.java:211)[396:org.apache.camel.camel-netty4-http:2.17.0.redhat-630254]

        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at org.apache.camel.component.netty4.http.handlers.HttpServerMultiplexChannelHandler.channelRead0(HttpServerMultiplexChannelHandler.java:113)[396:org.apache.camel.camel-netty4-http:2.17.0.redhat-630254]

        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:33)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:333)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:36)[358:io.netty.common:4.0.37.Final-redhat-2]

        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)[358:io.netty.common:4.0.37.Final-redhat-2]

Caused by: io.netty.channel.ChannelException: Failed to open a socket.

        at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:62)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:72)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at sun.reflect.GeneratedConstructorAccessor71.newInstance(Unknown Source)[:1.8.0_131]

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)[:1.8.0_131]

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)[:1.8.0_131]

        at java.lang.Class.newInstance(Class.java:442)[:1.8.0_131]

        at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:455)[360:io.netty.transport:4.0.37.Final-redhat-2]

        ... 38 more

Caused by: java.net.SocketException: Trop de fichiers ouverts

        at sun.nio.ch.Net.socket0(Native Method)[:1.8.0_131]

        at sun.nio.ch.Net.socket(Net.java:411)[:1.8.0_131]

        at sun.nio.ch.Net.socket(Net.java:404)[:1.8.0_131]

        at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:105)[:1.8.0_131]

        at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)[:1.8.0_131]

        at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:60)[360:io.netty.transport:4.0.37.Final-redhat-2]

        ... 44 more

2017-10-16 14:09:02,709 | WARN  | ttyServerTCPBoss | DefaultChannelPipeline           | 358 - io.netty.common - 4.0.37.Final-redhat-2 | An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

java.io.IOException: Trop de fichiers ouverts

        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)[:1.8.0_131]

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)[:1.8.0_131]

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)[:1.8.0_131]

        at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:135)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:69)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:564)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:505)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:419)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391)[360:io.netty.transport:4.0.37.Final-redhat-2]

        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)[358:io.netty.common:4.0.37.Final-redhat-2]

        at java.lang.Thread.run(Thread.java:748)[:1.8.0_131]

I'll really appreciate any help. The problem is blocking