Teiid performance

[yapnel]

Hi

I would like to know what is the recommended configuration for Teiid JVM when dealing with very large volume of data in the range of millions of records? Our instance has a mixed load of queries bring back large federated result set for reporting and small result sets with low latency response time for application.

We have 3GB of heap size as our default settings and we are getting fluctuation of the same query timing ranging from 2 seconds to 15 seconds for a relatively small query returning only 50 records. Further investigation, we have found the root cause was the full effect of GC pausing the world. Increasing the heap size to 8GB made the pausing even longer and not improving the performance at all.

Are you able to assist us in how to tune Teiid for better performance? We are on Teiid 7.4 EDS 5.2

Thanks

Nelson

[rareddy]

Nelson,

There are many resources on the web that suggest various settings to reduce the JVM GC pause times. In Teiid correctly allocate your batch sizes so that you are not holding on to too many rows that lead to big memory dumps. Check if there too many temp files being created in the "tmp/teiid/buffer" directory, this will indicate if your results being paged excessively or not.

If you are a customer please contact Redhat Technical support as they may have general JBoss AS JVM tuning guidelines to reduce the GC pauses.

EDS 5.3/ Teiid 7.7 has direct memory option, where you can configure Teiid to allocate direct memory instead of from heap, which is managed by Teiid directly for its purposes. This may also reduce some of the issues you are seeing. This version also auto calculates lot of tuple batch sizes according to the query it is executing.

Ramesh..

[shawkins]

> I would like to know what is the recommended configuration for Teiid JVM when dealing with very large volume of data in the range of millions of records?

If you're just returning forward-only results without joins/ordering/etc., then there shouldn't be much to configure speically for Teiid with regards to memory as we should be dealing with a bounded amount of data per request.  What is your max active plans setting? If that is too high, then you may be attempting to fullful too many requests concurrently.

You'll also want to check the hotspots for memory usage with a profiler is possible to see what is producing the most garbarge - Teiid buffering, the source drivers, etc.  If it's Teiid buffering then you'll want to target the buffer manager settings to improve the situation, which may include restricting the overall buffer memory available and possibly increasing the processing memory so that memory is allocated more on a query basis.

There have also been memory management improvements in every Teiid release.  For instance Teiid will more appropriately shape internal batches (and source batches with a recent 8.3 change) and you can move alot of the memory consumption off heap using direct memory starting with 7.6.

Steve