3 Replies Latest reply on Oct 30, 2015 8:27 AM by Vitaly Gorbunov

    Possible org.xnio.ByteBufferSlicePool memory leak in org.jboss.remoting3.remote.RemoteReadListener#handleEvent

    Vitaly Gorbunov Newbie

      I'm using wildfly 8.2.1 with patch from [EJBCLIENT-144] Synchronized connection pool singleton leads to performance bottleneck in case of slow connection to hos…

      To validate the patch I'm running arquillian stress test that tries to connect to other server in cycle with 10000 iteration using EJB scoped context and "remote.connection.myconnection.connect.timeout" equal to 1 (I can't share this test right now because it has many dependencies on our application code)

       

      As a result I'm getting OOM and memory leak report points to org.xnio.ByteBufferSlicePool.

      I've set the option -Djboss.remoting.debug-buffer-leaks=true and increased heap to avoid OOM and got the following logs:

      09:52:53,918 INFO  [org.jboss.remoting.remote] (Finalizer) Leaked a buffer which was allocated at: org.jboss.remoting3.remote.RemoteReadListener#handleEvent(RemoteReadListener.java:70) and used at:

          org.jboss.remoting3.remote.RemoteReadListener#handleEvent(RemoteReadListener.java:71)

          org.jboss.remoting3.remote.RemoteConnectionChannel#handleMessageData(RemoteConnectionChannel.java:440)

          org.jboss.remoting3.remote.InboundMessage#handleIncoming(InboundMessage.java:201)

          org.xnio.streams.BufferPipeInputStream#push(BufferPipeInputStream.java:80)

          org.xnio.streams.BufferPipeInputStream#read(BufferPipeInputStream.java:184)

          org.xnio.streams.BufferPipeInputStream#read(BufferPipeInputStream.java:133)

      After debugging I found that early return before this pool buffer could be used

       

      Pooled<ByteBuffer> pooled = connection.allocate(); //<--buffer is allocated
                  ByteBuffer buffer = pooled.getResource();
                  try {
                      for (;;) try {
                          synchronized (connection.getLock()) {
                              res = channel.receive(buffer);
                              if (res == -1) {
                                  log.trace("Received connection end-of-stream");
                                  try {
                                      channel.shutdownReads();
                                  } finally {
                                      handler.handleConnectionClose();
                                  }
                                  return;
                              } else if (res == 0) {
                                  log.trace("No message ready; returning");
                                  return; //<--here I get return and pooled buffer remains unclosed
                              }
                          }
      
      

       

      So I found the bug [REM3-200] Memory leak in org.xnio.ByteBufferSlicePool - JBoss Issue Tracker and it seems to me that it wasn't fixed completely.

       

      Update: I had a look at RemoteReadListener#handleEvent again and I see that I made a mistake - pooled buffer is closed in latest finally block indeed, somehow I missed it first time I looked. So now I have no idea why it was leaking.

       

      Workaround with -Djboss.remoting.pooled-buffers=false worked for me.