Monitoring Teiid (JMX?)

[markaddleman]

We are starting to performance test continuous executions.  Are monitoring hooks for key Teiid resources like:

• The number of in-use threads in the teiid-async pool?
• The queue length?

Also, what is the difference between the max-threads configuration here:

        <subsystem xmlns="urn:jboss:domain:threads:1.1">

            <bounded-queue-thread-pool name="teiid-async">

                <queue-length count="20000"/>

                <max-threads count="40"/>

            </bounded-queue-thread-pool>

        </subsystem>

and here:

            <max-threads>

                5000

            </max-threads>

?

btw, initial tests are positive.  Under a normal 512mb heap, tweaking the configuration (as you see above) and without really understanding what I'm doing, I can execute 5,000 continuous executions of the loopback translator, each returning 1,000,000 rows.  CPU is pretty low and the application gets very high throughput of results.

More typical result sets for us will be around 10-100 rows and at ~2,500 CEs, we get messages like

Uncaught exception processing work: java.util.concurrent.RejectedExecutionException: Task org.teiid.dqp.internal.process.ThreadReuseExecutor$3@d7e3770 rejected from org.teiid.dqp.internal.process.ThreadReuseExecutor$2@1a5c2b39[Running, pool size = 4061, active threads = 3800, queued tasks = 0, completed tasks = 808285]

I'm not sure how to match the stats in the log message to configuration settings.  Regardless, we're maxing out something.  Would that be because each execution ends up back on the queue more quickly (due to small result set size and very quick turn-around by the translator)?  I'm trying to put together my mental model of how Teiid manages its work. 

I'm still putting together exactly how we'll use CEs in our application, with no effort put into smartly managing the queries, I don't think we'll have more than 2,500.  With a bit of smarts in our UI, we can probably drop that to fewer than 500.  At this stage, I'm trying to get a feel for just how much effort we need to put into building those smarts.

[rareddy]

Mark,

There is no JMX support Teiid does on its own, however AS7 does expose JMX for monitoring various core components. We do expose lot of Teiid specifics over the CLI. Not  given the JMX much thought in this version.

teiid-async = this is used for any asynchronous jobs like metadata load (uses the AS7 thread subsystem, available through JMX)

max-threads = max processing and connector threads combined. (This does not use AS7 thread subsystem, as this does not do proactive re-use of idle threads until minimum thread-count is reached)

You are actually limited by "max-active-plans" for max concurrent executing queries. The default is 20. Look at jboss-teiid.xsd file for all the configuration parameters in the "teiid-jboss-integration" project.

I see the above exception is only thrown when there are active jobs or client is still submitting jobs and server is being shutdown, otherwise all the excess jobs do get queued. For resultset sizes you have the default batch sizes are fine.

Ramesh..

[markaddleman]

JMX support isn't all that important to me.  I can probably get what I need through the CLI.

Thanks for the pointer to to the docs and max-active-plans.  What constitutes an active plan?  My test sends a constant SQL string, "select * from LoopTest.test LIMIT 100"  I can see how 20 or more of these are executed in parallel but is that one plan or twenty?

[shawkins]

maxActivePlans are not counted against a continous execution.  The only time we throw a rejectedexecutionexception directly from our logic is when the the thread pool has been terminated.  So you've actually hit a lower level limit.