Fuse ESB Cluster generate "too many files opened"
fsandreau Oct 19, 2017 12:07 PMDear all,
We have a very complex problem using FUSE ESB 6.3 redhat-630254. For some reasons, on each cluster's node, the number of file descriptor used exceed the limit (more than 65000).
Our architecture is based on 5 nodes with the bundle service deployed on each one (using Fabric profile).
We are using :
camel vesion 2.17.0.redhat-630187
camel-netty4-http version 2.17.0.redhat-630187
Below the exception :
2017-10-16 14:09:02,577 | ERROR | entExecutorGroup | DefaultErrorHandler | 378 - org.apache.camel.camel-core - 2.17.0.redhat-630254 | Failed delivery for (MessageId: ID-SVR-ESB-FUSE-SLAVE-3-49950-1508016636199-4-2511736 on ExchangeId: ID-SVR-ESB-FUSE-SLAVE-3-49950-1508016636199-4-2511737). Exhausted after delivery attempt: 1 caught: io.netty.channel.ChannelException: Unable to create Channel from class class io.netty.channel.socket.nio.NioSocketChannel. Processed by failure processor: FatalFallbackErrorHandler[Pipeline[[Channel[SetHeader(CamelHttpResponseCode, 500)], Channel[SetHeader(Content-Type, application/json)], Channel[DelegateSync[com.ldlc.commons.technical.esb.route.CommonsExceptionRoute$$Lambda$18/279035000@639d36d6]], Channel[Marshal[org.apache.camel.component.jackson.JacksonDataFormat@3f4fd890]]]]]n | n | Message Historyn | ---------------------------------------------------------------------------------------------------------------------------------------n | RouteId ProcessorId Processor Elapsed (ms)n | [route78 ] [route78 ] [http://0.0.0.0:10009/api/rest/Configuration/user/%7BsiteId%7D/%7BuserId%7D?htt] [ 31]n | [route78 ] [restBinding64 ] [ ] [ 0]n | [route78 ] [route78 ] [direct:configurationService.getByUserId ] [ 31]n | [route95 ] [removeHeader75 ] [removeHeader[CamelHttpPath] ] [ 0]n | [route95 ] [configurationBackE] [ ] [ 30]n | [ ] [setHeader327 ] [setHeader[CamelHttpResponseCode] ] [ 0]n | [ ] [setHeader328 ] [setHeader[Content-Type] ] [ 0]n | [ ] [process133 ] [Processor@0x639d36d6 ] [ 27]n | [ ] [marshal140 ] [marshal[org.apache.camel.model.dataformat.JsonDataFormat@76c98c8b] ] [ 2]n | n | Stacktracen | ---------------------------------------------------------------------------------------------------------------------------------------
io.netty.channel.ChannelException: Unable to create Channel from class class io.netty.channel.socket.nio.NioSocketChannel
at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:457)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:308)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.bootstrap.Bootstrap.doConnect(Bootstrap.java:135)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:117)[360:io.netty.transport:4.0.37.Final-redhat-2]
at org.apache.camel.component.netty4.NettyProducer.openConnection(NettyProducer.java:420)[395:org.apache.camel.camel-netty4:2.17.0.redhat-630254]
at org.apache.camel.component.netty4.NettyProducer$NettyProducerPoolableObjectFactory.makeObject(NettyProducer.java:550)[395:org.apache.camel.camel-netty4:2.17.0.redhat-630254]
at org.apache.camel.component.netty4.NettyProducer$NettyProducerPoolableObjectFactory.makeObject(NettyProducer.java:546)[395:org.apache.camel.camel-netty4:2.17.0.redhat-630254]
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)[412:org.apache.commons.pool:1.6.0.redhat-9]
at org.apache.camel.component.netty4.NettyProducer.process(NettyProducer.java:206)[395:org.apache.camel.camel-netty4:2.17.0.redhat-630254]
at org.apache.camel.component.netty4.http.NettyHttpProducer.process(NettyHttpProducer.java:52)[396:org.apache.camel.camel-netty4-http:2.17.0.redhat-630254]
at org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:124)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:341)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:119)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:196)[378:org.apache.camel.camel-core:2.17.0.redhat-630254]
at org.apache.camel.component.netty4.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:138)[395:org.apache.camel.camel-netty4:2.17.0.redhat-630254]
at org.apache.camel.component.netty4.handlers.ServerChannelHandler.channelRead0(ServerChannelHandler.java:109)[395:org.apache.camel.camel-netty4:2.17.0.redhat-630254]
at org.apache.camel.component.netty4.http.handlers.HttpServerChannelHandler.channelRead0(HttpServerChannelHandler.java:211)[396:org.apache.camel.camel-netty4-http:2.17.0.redhat-630254]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)[360:io.netty.transport:4.0.37.Final-redhat-2]
at org.apache.camel.component.netty4.http.handlers.HttpServerMultiplexChannelHandler.channelRead0(HttpServerMultiplexChannelHandler.java:113)[396:org.apache.camel.camel-netty4-http:2.17.0.redhat-630254]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:33)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:333)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:36)[358:io.netty.common:4.0.37.Final-redhat-2]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)[358:io.netty.common:4.0.37.Final-redhat-2]
Caused by: io.netty.channel.ChannelException: Failed to open a socket.
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:62)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:72)[360:io.netty.transport:4.0.37.Final-redhat-2]
at sun.reflect.GeneratedConstructorAccessor71.newInstance(Unknown Source)[:1.8.0_131]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)[:1.8.0_131]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)[:1.8.0_131]
at java.lang.Class.newInstance(Class.java:442)[:1.8.0_131]
at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:455)[360:io.netty.transport:4.0.37.Final-redhat-2]
... 38 more
Caused by: java.net.SocketException: Trop de fichiers ouverts
at sun.nio.ch.Net.socket0(Native Method)[:1.8.0_131]
at sun.nio.ch.Net.socket(Net.java:411)[:1.8.0_131]
at sun.nio.ch.Net.socket(Net.java:404)[:1.8.0_131]
at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:105)[:1.8.0_131]
at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)[:1.8.0_131]
at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:60)[360:io.netty.transport:4.0.37.Final-redhat-2]
... 44 more
2017-10-16 14:09:02,709 | WARN | ttyServerTCPBoss | DefaultChannelPipeline | 358 - io.netty.common - 4.0.37.Final-redhat-2 | An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Trop de fichiers ouverts
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)[:1.8.0_131]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)[:1.8.0_131]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)[:1.8.0_131]
at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:135)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:69)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:564)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:505)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:419)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391)[360:io.netty.transport:4.0.37.Final-redhat-2]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)[358:io.netty.common:4.0.37.Final-redhat-2]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_131]
I'll really appreciate any help. The problem is blocking