Out of memory exception

[sonalm13]

Hi,

I have started 3 nodes in an infinispan cluster. I left the cluster nodes running over the weekend. Today we observed following error:

ERROR [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-69) ISPN005009: Unexpected error before any request parameters read: java.lang.OutOfMemoryError: Direct buffer memory

  at java.nio.Bits.reserveMemory(Bits.java:658) [rt.jar:1.7.0_40]

  at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) [rt.jar:1.7.0_40]

  at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) [rt.jar:1.7.0_40]

  at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174) [rt.jar:1.7.0_40]

  at sun.nio.ch.IOUtil.write(IOUtil.java:58) [rt.jar:1.7.0_40]

  at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) [rt.jar:1.7.0_40]

  at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:202) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:147) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.Channels.write(Channels.java:725) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.Channels.write(Channels.java:704) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.Channels.write(Channels.java:671) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248) [netty-3.6.6.Final.jar:]

  at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:163) [infinispan-server-core-6.0.0.Final.jar:6.0.0.Final]

  at org.infinispan.server.core.AbstractProtocolDecoder.decodeKey(AbstractProtocolDecoder.scala:99) [infinispan-server-core-6.0.0.Final.jar:6.0.0.Final]

  at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:53) [infinispan-server-core-6.0.0.Final.jar:6.0.0.Final]

  at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:29) [infinispan-server-core-6.0.0.Final.jar:6.0.0.Final]

  at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.6.6.Final.jar:]

  at org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:377) [infinispan-server-core-6.0.0.Final.jar:6.0.0.Final]

  at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90) [netty-3.6.6.Final.jar:]

  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.6.6.Final.jar:]

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_40]

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_40]

  at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_40]

Any suggestions regarding this will be really helpful.

Thanks,

Sonal

[wdfink]

The "direct memory buffer" reside outside of the heap memory.

I see several possible issues:

- you run the instances on the same machine and JVM's+system collect all the memory and the process can not allocate more

- there is a memory leak (nio netty infinispan ?) which occupy the memory if you running a longer time

You need to describe the environment and investigate a bit deeper

[sonalm13]

I have 3 nodes running in infinispan cluster with each node having its file store for persistence. Basically we are refreshing cache every one hour with data persisted in cache store as well.

you run the instances on the same machine and JVM's+system collect all the memory and the process can not allocate more

We don't think this should be the case as we have enough memory, also we don't have much data data in the cache as we are still evaluating infinispan.

there is a memory leak (nio netty infinispan ?) which occupy the memory if you running a longer time

I saw this https://github.com/netty/netty/issues/1574 . Is this the possibility for above mentioned error. If yes, how can we have a workaround for this or when will its solution be available with infiinispan server modules?

Also, is there any fail-over mechanism when one of the nodes fail(like here) request get transferred to other nodes?