Full gc is constantly being performed
cosmin_go May 9, 2017 10:57 AMWe have an infinispan 7.0.3 instance running that performs unnecessary garbage collections and the infinispan client terminates with this exception:
ERROR [RetryOnFailureOperation] (EJB default - 29) ISPN004007: Exception encountered. Retry 10 out of 10: org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException
On the infinispan server.log we have this exception:
2017-05-03 09:53:07,288 WARNING [io.netty.channel.AbstractChannel] (HotRodServerMaster-6-1) Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xebdfa00d, /127.0.0.1:16094 => /127.0.0.2:11222]: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_51]
at java.lang.Thread.start(Thread.java:713) [rt.jar:1.7.0_51]
at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:843)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:717)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:416)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:60)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:48)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:64)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:252)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:332)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:318)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:92)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) [netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)[netty-all-4.0.20.Final.jar:4.0.20.Final]
at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
The infinispan is running on a SunOS 5.11 11.3 sun4v sparc sun4v machine.
Our java version is :
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
I've also attached the detailed gc log.
Any hints are appreciated.
-
2017-05-08_10_52_06-gc.log.zip 30.5 KB