We are starting to performance test continuous executions. Are monitoring hooks for key Teiid resources like:
- The number of in-use threads in the teiid-async pool?
- The queue length?
Also, what is the difference between the max-threads configuration here:
btw, initial tests are positive. Under a normal 512mb heap, tweaking the configuration (as you see above) and without really understanding what I'm doing, I can execute 5,000 continuous executions of the loopback translator, each returning 1,000,000 rows. CPU is pretty low and the application gets very high throughput of results.
More typical result sets for us will be around 10-100 rows and at ~2,500 CEs, we get messages like
Uncaught exception processing work: java.util.concurrent.RejectedExecutionException: Task org.teiid.dqp.internal.process.ThreadReuseExecutor$3@d7e3770 rejected from org.teiid.dqp.internal.process.ThreadReuseExecutor$2@1a5c2b39[Running, pool size = 4061, active threads = 3800, queued tasks = 0, completed tasks = 808285]
I'm not sure how to match the stats in the log message to configuration settings. Regardless, we're maxing out something. Would that be because each execution ends up back on the queue more quickly (due to small result set size and very quick turn-around by the translator)? I'm trying to put together my mental model of how Teiid manages its work.
I'm still putting together exactly how we'll use CEs in our application, with no effort put into smartly managing the queries, I don't think we'll have more than 2,500. With a bit of smarts in our UI, we can probably drop that to fewer than 500. At this stage, I'm trying to get a feel for just how much effort we need to put into building those smarts.