1 2 Previous Next 24 Replies Latest reply on May 19, 2017 9:17 AM by cosmin_go

    Full gc is constantly being performed

    cosmin_go

      We have an infinispan 7.0.3 instance running that performs unnecessary garbage collections and the infinispan client terminates with this exception:

      ERROR [RetryOnFailureOperation] (EJB default - 29) ISPN004007: Exception encountered. Retry 10 out of 10: org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException

       

      On the infinispan server.log we have this exception:

      2017-05-03 09:53:07,288 WARNING [io.netty.channel.AbstractChannel] (HotRodServerMaster-6-1) Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xebdfa00d, /127.0.0.1:16094 => /127.0.0.2:11222]: java.lang.OutOfMemoryError: unable to create new native thread
               at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_51]
               at java.lang.Thread.start(Thread.java:713) [rt.jar:1.7.0_51]
               at io.netty.util.concurrent.SingleThreadEventExecutor.startThread(SingleThreadEventExecutor.java:843)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:717)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:416)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:60)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:48)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:64)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:252)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:332)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:318)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:92)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) [netty-all-4.0.20.Final.jar:4.0.20.Final]
               at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)[netty-all-4.0.20.Final.jar:4.0.20.Final]
               at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

       

      The infinispan is running on a SunOS 5.11 11.3 sun4v sparc sun4v machine.

      Our java version is :

      java version "1.8.0_121"

      Java(TM) SE Runtime Environment (build 1.8.0_121-b13)

      Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

       

      I've also attached the detailed gc log.

       

      Any hints are appreciated.

        1 2 Previous Next