0 Replies Latest reply on Sep 14, 2015 7:03 AM by Antoine Herzog

    Too many server instances : "java.lang.OutOfMemoryError: unable to create new native thread"

    Antoine Herzog Master

      Hi,

       

      This is info, to help if someone has the same problem.

       

      I launch two server groups, with each having two instances of server, in domain mode, with full-ha profile for cluster.

      On the same portable PC.

      That makes 4 instances of servers.

       

      When the third instance starts, this error occurs :

       

      [Server:server-two] 12:28:48,586 ERROR [org.hornetq.core.client] (Thread-6 (HornetQ-server-HornetQServerImpl::serverUUID=7e000005-56fa-11e5-b064-f538ff1a553f-1610892918)) HQ214017: Caught unexpected Throwable: java.lang.OutOfMemoryError: unable to create new native thread

      [Server:server-two]     at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_25]

      [Server:server-two]     at java.lang.Thread.start(Thread.java:693) [rt.jar:1.7.0_25]

      [Server:server-two]     at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) [rt.jar:1.7.0_25]

      [Server:server-two]     at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590) [rt.jar:1.7.0_25]

      [Server:server-two]     at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333) [rt.jar:1.7.0_25]

      [Server:server-two]     at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594) [rt.jar:1.7.0_25]

      [Server:server-two]     at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1416) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1121) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:266) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:881) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:669) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:653) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ServerLocatorImpl$3.run(ServerLocatorImpl.java:631) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]

      [Server:server-two]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]

      [Server:server-two]     at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]

      [Server:server-two]

      [Server:server-two] 12:28:48,590 WARNING [io.netty.channel.AbstractChannel] (Thread-6 (HornetQ-server-HornetQServerImpl::serverUUID=7e000005-56fa-11e5-b064-f538ff1a553f-1610892918)) Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xd08bbe83]: java.lang.OutOfMemoryError: unable to create new native thread

      [Server:server-two]     at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_25]

      [Server:server-two]     at java.lang.Thread.start(Thread.java:693) [rt.jar:1.7.0_25]

      [Server:server-two]     at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:812) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:690) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:415) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:60) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:48) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:64) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:305) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.bootstrap.Bootstrap.doConnect(Bootstrap.java:133) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:115) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:715) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1275) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1121) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:266) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:881) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:669) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:653) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ServerLocatorImpl$4.run(ServerLocatorImpl.java:1699) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]

      [Server:server-two]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]

      [Server:server-two]     at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]

      [Server:server-two]

      [Server:server-two] 12:28:48,592 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614)) Exception in thread "Thread-0 (HornetQ-client-global-scheduled-threads-1807040614)" java.lang.OutOfMemoryError: unable to create new native thread

      [Server:server-two] 12:28:48,595 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614))     at java.lang.Thread.start0(Native Method)

      [Server:server-two] 12:28:48,596 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614))     at java.lang.Thread.start(Thread.java:693)

      [Server:server-two] 12:28:48,596 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614))     at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)

      [Server:server-two] 12:28:48,596 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614))     at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)

      [Server:server-two] 12:28:48,597 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614))     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)

      [Server:server-two] 12:28:48,597 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614))     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

      [Server:server-two] 12:28:48,597 ERROR [stderr] (Thread-0 (HornetQ-client-global-scheduled-threads-1807040614))     at java.lang.Thread.run(Thread.java:724)

      [Server:server-two] 12:28:48,607 ERROR [org.hornetq.core.client] (Thread-6 (HornetQ-server-HornetQServerImpl::serverUUID=7e000005-56fa-11e5-b064-f538ff1a553f-1610892918)) HQ214016: Failed to create netty connection: java.lang.OutOfMemoryError: unable to create new native thread

      [Server:server-two]     at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_25]

      [Server:server-two]     at java.lang.Thread.start(Thread.java:693) [rt.jar:1.7.0_25]

      [Server:server-two]     at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:812) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:690) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:415) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:60) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:48) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:64) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:305) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.bootstrap.Bootstrap.doConnect(Bootstrap.java:133) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:115) [netty-all-4.0.15.Final.jar:4.0.15.Final]

      [Server:server-two]     at org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:715) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1275) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1121) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:266) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:881) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:669) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:653) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.core.client.impl.ServerLocatorImpl$4.run(ServerLocatorImpl.java:1699) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104) [hornetq-core-client-2.4.5.Final.jar:]

      [Server:server-two]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]

      [Server:server-two]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]

      [Server:server-two]     at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]

      [Server:server-two]

      [Server:server-three] 12:28:48,640 INFO  [org.jboss.ws.common.management] (MSC service thread 1-12) JBWS022052: Starting JBoss Web Services - Stack CXF Server 4.3.2.Final

       

      It needs to increase the limit of processes allowed for each user, by the OS.

      ulimits gives :

      $ ulimit -Su 
      1024
      $ ulimit -Hu
      61459
      

       

      Setting the number of processes up to 2048 is enough.

       

      For that, in the file : /etc/security/limits.d/90-nproc.conf

      set :

      *     soft     nproc     2048 ! !@
      

       

      Then the server starts fine,

       

      Hope it helps,

      Antoine

      JBoss Portal and GateIn, WildFly, EPP, EAP, Exo Platform, JSF, Richfaces, J2EE.

      http://www.sysemo.com/Sysemo-expertise-portails-jboss-portal.php