-
1. Re: buffer manager
> Could you please enlighten me under what conditions/operations the buffer manager will start to page data into the file system? I.e. large joins etc....
Any operation that involves row buffering or the creation of temporary tables can eventually force the usage of disk space. The most common memory intensive operation is a full sort, which can be used for join, grouping, and order by operations. The buffer manager has several memory layers. It first makes an estimate of the heap space in use by current processing and batches held in memory. If this exceeds the desired bytes that are "reserved" for the buffer manager's use, then the batches will be serialized out to a fixed memory buffer. The fixed memory buffer (governed by the max memory buffer size setting) in turn acts as a caching layer that based upon memory pressure will evict to disk.
> Is there a max size on the JVM heap size that will start causing issues to teiid?
You'll want to be specific if there is an exact scenario that you are experiencing. The interaction with the heap can be complicated. First the initial size estimation by the buffer manager of the heap usage is not exact (as runtime memory usage calculations in java tend to be). There are several properties that can help tune the first level usage, such as the processing kb (which can dedicate more or less memory on a per operation basis) and the overall reserved kb. As batches are swapped out into the fixed memory buffer we typically retain a reference (a soft reference for caching that prefers memory) to recover the batch if reused before a full gc is performed. The buffer manager does not ensure that the collection actually takes place though and effective assumes that by making the batch gc eligible that it should no longer be counted against the reserve (maintaining a reference queue for this purpose seems to add overhead without much benefit). Also if the fixed memory buffer is a large size this may still incur a significant amount of gc overhead. In which case the usage of the direct memory option can help as it will move the fixed memory buffer off heap.