Teiid queries not never releasing the memory from heap resulting Garbage collection failures

[vishnusv]

We have a federated VDB with a Views made from attributes from multiple Oracle data sources. The View results in 2 million records and it involves joining tables with 2 million records from two different data sources(please see example view below). We have several such views. Our Heapsize for the JVM is 24GB.

When we fetch data from these views, the memory keeps growing in heap. The memory never releases back and eventually in a day or two, it reaches the max heap size and at some time results in the following error java.lang.OutOfMemoryError: GC overhead limit exceeded.(Attached HeapDump screenshot after I forced GC, I can share the HeapDump if required)

It appears the memory is not getting released. We haven't explicity enabled any resultset caching and set the default Teiid buffer settings. After we encountered this error, we have played with Teiid buffer sizes like max reserve kb and max memory buffer, but haven't seen much luck .  Trying to find out what settings should we be changing in this scenario and why the memory is not getting released.

Example view view1. All the source tables have more than 2 million records.

CREATE VIEW VIEW1

(

C1 bigdecimal(32767),

C2 bigdecimal(32767),

C3 bigdecimal(32767),

C4 bigdecimal(32767),

C5 string(4000),

C6 string(4000),

C7 string(4000),

   CONSTRAINT VIEW1_PK PRIMARY KEY (C1)

)

AS

SELECT DISTINCT

SRC1T1 .C1,

SRC2T1.C2,

SRC1T1.C3,

SRC1T1.C4,

SRC3T1.C5,

SRC3T1.C6,

SRC2T1.C7

FROM

SRC1.TBL1 SRC1T1 INNER JOIN SRC2.TBL1 SRC2TBL1 ON SRC1T1.C1 = SRC1TBL1.C1

INNER JOIN SRC3.TBL1  SRC3TBL1 ON SRC1TBL1.C2 = SRC3TBL1.C2

;

Heap Dump screenshot:

[shawkins]

Is this the same scenario as Teiid is throwing java.lang.OutOfMemoryError while loading maximum data. ?

> It appears the memory is not getting released.

There are several memory regions for Teiid.  There is a fixed byte buffer region that once allocated will not be garbage collected - Memory Management · GitBook You should expect that to not be released.

> We haven't explicity enabled any resultset caching and set the default Teiid buffer settings.

Additional on heap memory usage from query processing or caching is either paged to disk as needed or released when the query is finished.

> After we encountered this error, we have played with Teiid buffer sizes like max reserve kb and max memory buffer, but haven't seen much luck . 

Yes, you'll need to fully understand what is happening before trying to alter the configuration.

> Attached HeapDump screenshot after I forced GC, I can share the HeapDump if required

The best would be a similar screenshot at the time the gc error occurs - using the HeapDumpOnOutOfMemoryError option.

[vishnusv]

Steven,

>> Is this the same scenario as Teiid is throwing java.lang.OutOfMemoryError while loading maximum data. ?

  The request seems similar. We also have large source tables with millions of records, though we are only interested in reads

I got the heap dumps . It appears the BufferManagerImpl is holding 18GB in heap. All our Teiid buffer manager settings are set to default. with our JVM  heap at 24 GB and machine (32GB).

[shawkins]

What version are you using?  It looks like the memory being held is on heap storage of batches which should be related to the current query that is processing - that definitely is not expected as memory pressure should force stuff to disk.

[vishnusv]

Steven,

We are using Teiid 9.1.1. Is there any known issues with this version?.

The observation we have seen is, after the query gets completed, the memory is not released. On executing a second view, the memory is accumulated.

[shawkins]

Yes it's quite possible that you are hitting an issue that has already been addressed.  Can you redo your scenario on a later version?  If there is an issue that requires a code fix for the community releases that will only be applied to 12.x.

[vishnusv]

Steven,

I may not be able to validate with latest version of Teiid immediately. Since we had left the Teiid buffer settings unchanged at default, will changing any of the parameters restrict the usage of Teiid memory buffer?.

You mentioned, There is a fixed byte buffer region that once allocated will not be garbage collected. Is this memory-buffer-space?. Curious to know, What is being held in fixed buffer space that can't be garbage collected? How is memory-buffer-space different from max reserve kb?.

Thanks,

Vishnu

[shawkins]

> Since we had left the Teiid buffer settings unchanged at default, will changing any of the parameters restrict the usage of Teiid memory buffer?

The closest would be the max reserve kb setting, see below for more.

> You mentioned, There is a fixed byte buffer region that once allocated will not be garbage collected. Is this memory-buffer-space?

Yes.

> Curious to know, What is being held in fixed buffer space that can't be garbage collected?

They are large bytebuffers that would frequently be reallocated if they were allowed to be garbage collected.  They can also be allocated off heap as well. 

However the fixed region is not your issue as it's live on heap batches that are being held in your memory snapshot above.

> How is memory-buffer-space different from max reserve kb?.

max reserve kb refers to approximately how much on heap memory in the form of live batches can be held by the buffer manager.  You may try manually setting it - even to just a 0 value to determine if your scenario will then succeed.  See Memory Management · GitBook - which has updated property names, or an older version Memory Management · GitBook for more.  The section called Common Configuration Scenarios at the end discusses restricting heap usage.

[skethire]

After more analysis, we noticed that this behavior occurs when there is a DISTINCT clause in the federated view.

The federated view has joins from tables from 3 or 4 different data sources.

Do you know of any issue with processing DISTINCT?

[shawkins]

> Do you know of any issue with processing DISTINCT?

None specifically.  If Teiid processes a distinct operation, it can be quite expensive for large data sets.  There have been several refinements since 9.1.1 in memory management and multi-pass sorts.