2 Replies Latest reply on Dec 9, 2013 7:55 AM by Sonal Maheshwari

    Out of memory exception

    Sonal Maheshwari Newbie

      Hi,

       

      I have started 3 nodes in an infinispan cluster. I left the cluster nodes running over the weekend. Today we observed following error:

       

      ERROR [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-69) ISPN005009: Unexpected error before any request parameters read: java.lang.OutOfMemoryError: Direct buffer memory

        at java.nio.Bits.reserveMemory(Bits.java:658) [rt.jar:1.7.0_40]

        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) [rt.jar:1.7.0_40]

        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) [rt.jar:1.7.0_40]

        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174) [rt.jar:1.7.0_40]

        at sun.nio.ch.IOUtil.write(IOUtil.java:58) [rt.jar:1.7.0_40]

        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) [rt.jar:1.7.0_40]

        at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:202) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:147) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.Channels.write(Channels.java:725) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.Channels.write(Channels.java:704) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.Channels.write(Channels.java:671) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248) [netty-3.6.6.Final.jar:]

        at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:163) [infinispan-server-core-6.0.0.Final.jar:6.0.0.Final]

        at org.infinispan.server.core.AbstractProtocolDecoder.decodeKey(AbstractProtocolDecoder.scala:99) [infinispan-server-core-6.0.0.Final.jar:6.0.0.Final]

        at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:53) [infinispan-server-core-6.0.0.Final.jar:6.0.0.Final]

        at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:29) [infinispan-server-core-6.0.0.Final.jar:6.0.0.Final]

        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.6.6.Final.jar:]

        at org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:377) [infinispan-server-core-6.0.0.Final.jar:6.0.0.Final]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90) [netty-3.6.6.Final.jar:]

        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.6.6.Final.jar:]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_40]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_40]

        at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_40]

       

      Any suggestions regarding this will be really helpful.

       

      Thanks,

      Sonal

        • 1. Re: Out of memory exception
          Wolf-Dieter Fink Master

          The "direct memory buffer" reside outside of the heap memory.

          I see several possible issues:

          - you run the instances on the same machine and JVM's+system collect all the memory and the process can not allocate more

          - there is a memory leak (nio netty infinispan ?) which occupy the memory if you running a longer time

           

          You need to describe the environment and investigate a bit deeper

          • 2. Re: Out of memory exception
            Sonal Maheshwari Newbie

            I have 3 nodes running in infinispan cluster with each node having its file store for persistence. Basically we are refreshing cache every one hour with data persisted in cache store as well.

             

            you run the instances on the same machine and JVM's+system collect all the memory and the process can not allocate more

             

            We don't think this should be the case as we have enough memory, also we don't have much data data in the cache as we are still evaluating infinispan.

             

            there is a memory leak (nio netty infinispan ?) which occupy the memory if you running a longer time

             

            I saw this https://github.com/netty/netty/issues/1574 . Is this the possibility for above mentioned error. If yes, how can we have a workaround for this or when will its solution be available with infiinispan server modules?

             

            Also, is there any fail-over mechanism when one of the nodes fail(like here) request get transferred to other nodes?