3 Replies Latest reply on Nov 5, 2015 3:32 PM by Steven Hawkins

    BufferManagerImpl getting into inconsistent state and causing eventual OOM with Teiid

    Nick Ross Newbie

      Hi


      We have been experiencing some sporadic OOM issues with Teiid and have hit a brickwall with our investigations. The BufferManagerImpl seems to get into an inconsistent state and stops evicting data - eventually leading to OOM.


      Monitoring via debugging and some additional logging the memoryEntries and eviction queues seem to behave as expected (growing then shrinking back down over time). However, during occasional periods of heavy load the BufferManagerImpl gets into an inconsistent state, once in this state evictions stop and BufferManagerImpl#memoryEntries are never cleared - even once all querying has completed.

       

      Some observations once the inconsistent state is reached;

      • This state is reached before the heap is put under pressure
      • The inconsistent state is never resolved, even when querying has ceased.
      • BufferManagerImpl#memoryEntries does not clear out, refs are strongly reachable in heap dump
      • The BufferManagerImpl#Cleaner timer is still running
      • Both BufferManagerImpl#evictionQueue and BufferManager#initialEvictionQueue collections are empty
      • BufferManagerImpl#evictionQueue.getSize() and BufferManager#initialEvictionQueue.getSize() report negative sizes (LrfuEvictionQueue)

       

      Are there any known issues in this area?
      Any suggestions for further troubleshooting?

      This is of course hard to repeat so raising a Jira with a reproducible use case may not be feasible.

      I can provide further info if required, heap dump screenshots etc...

       

      Config: v8.10.0, 8Gb heap, no disk cache, 10 query plans, default mem settings.


      Thanks,

      -Nick