Too many server instances : "java.lang.OutOfMemoryError: unable to create new native thread"
antoine_h Sep 14, 2015 7:03 AMHi,
This is info, to help if someone has the same problem.
I launch two server groups, with each having two instances of server, in domain mode, with full-ha profile for cluster.
On the same portable PC.
That makes 4 instances of servers.
When the third instance starts, this error occurs :
[Server:server-two] 12:28:48,586 ERROR [org.hornetq.core.client] (Thread-6 (HornetQ-server-HornetQServerImpl::serverUUID=7e000005-56fa-11e5-b064-f538ff1a553f-1610892918)) HQ214017: Caught unexpected Throwable: java.lang.OutOfMemoryError: unable to create new native thread
[Server:server-two] at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_25]
[Server:server-two] at java.lang.Thread.start(Thread.java:693) [rt.jar:1.7.0_25]
[Server:server-two] at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) [rt.jar:1.7.0_25]
[Server:server-two] at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590) [rt.jar:1.7.0_25]
[Server:server-two] at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333) [rt.jar:1.7.0_25]
[Server:server-two] at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594) [rt.jar:1.7.0_25]
[Server:server-two] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1416) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1121) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:266) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:881) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:669) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:653) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ServerLocatorImpl$3.run(ServerLocatorImpl.java:631) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
[Server:server-two] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
[Server:server-two] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
[Server:server-two]
[Server:server-two] 12:28:48,590 WARNING [io.netty.channel.AbstractChannel] (Thread-6 (HornetQ-server-HornetQServerImpl::serverUUID=7e000005-56fa-11e5-b064-f538ff1a553f-1610892918)) Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xd08bbe83]: java.lang.OutOfMemoryError: unable to create new native thread
[Server:server-two] at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_25]
[Server:server-two] at java.lang.Thread.start(Thread.java:693) [rt.jar:1.7.0_25]
[Server:server-two] at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:812) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:690) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:415) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:60) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:48) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:64) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:305) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.bootstrap.Bootstrap.doConnect(Bootstrap.java:133) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:115) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:715) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1275) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1121) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:266) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:881) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:669) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:653) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ServerLocatorImpl$4.run(ServerLocatorImpl.java:1699) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
[Server:server-two] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
[Server:server-two] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
[Server:server-two]
[Server:server-two] 12:28:48,592 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614)) Exception in thread "Thread-0 (HornetQ-client-global-scheduled-threads-1807040614)" java.lang.OutOfMemoryError: unable to create new native thread
[Server:server-two] 12:28:48,595 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614)) at java.lang.Thread.start0(Native Method)
[Server:server-two] 12:28:48,596 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614)) at java.lang.Thread.start(Thread.java:693)
[Server:server-two] 12:28:48,596 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614)) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
[Server:server-two] 12:28:48,596 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614)) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
[Server:server-two] 12:28:48,597 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614)) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
[Server:server-two] 12:28:48,597 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614)) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[Server:server-two] 12:28:48,597 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614)) at java.lang.Thread.run(Thread.java:724)
[Server:server-two] 12:28:48,607 ERROR [org.hornetq.core.client] (Thread-6 (HornetQ-server-HornetQServerImpl::serverUUID=7e000005-56fa-11e5-b064-f538ff1a553f-1610892918)) HQ214016: Failed to create netty connection: java.lang.OutOfMemoryError: unable to create new native thread
[Server:server-two] at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_25]
[Server:server-two] at java.lang.Thread.start(Thread.java:693) [rt.jar:1.7.0_25]
[Server:server-two] at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:812) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:690) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:415) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:60) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:48) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:64) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:305) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.bootstrap.Bootstrap.doConnect(Bootstrap.java:133) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:115) [netty-all-4.0.15.Final.jar:4.0.15.Final]
[Server:server-two] at org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:715) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1275) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1121) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:266) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:881) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:669) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:653) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.core.client.impl.ServerLocatorImpl$4.run(ServerLocatorImpl.java:1699) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104) [hornetq-core-client-2.4.5.Final.jar:]
[Server:server-two] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
[Server:server-two] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
[Server:server-two] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
[Server:server-two]
[Server:server-three] 12:28:48,640 INFO [org.jboss.ws.common.management] (MSC service thread 1-12) JBWS022052: Starting JBoss Web Services - Stack CXF Server 4.3.2.Final
It needs to increase the limit of processes allowed for each user, by the OS.
ulimits gives :
$ ulimit -Su 1024 $ ulimit -Hu 61459
Setting the number of processes up to 2048 is enough.
For that, in the file : /etc/security/limits.d/90-nproc.conf
set :
* soft nproc 2048 ! !@
Then the server starts fine,
Hope it helps,
Antoine
JBoss Portal and GateIn, WildFly, EPP, EAP, Exo Platform, JSF, Richfaces, J2EE.
http://www.sysemo.com/Sysemo-expertise-portails-jboss-portal.php