org.teiid.common.buffer.BlockedException

[nsabina]

We are getting high CPU usage for continuous queries (any demo object). As observed CPU usage is growing over time as user clicks on the object and keep watching the page being autorefreshed.

For the purpose of my test I set autorefresh interval to one second. In debug mode I am seeing following exceptions in the log (larger log fragment is attached). There is a set of 4 or 5 of them for every async execution.

06 Jan 2013 22:39:20,462 PST DEBUG [org.teiid.COMMAND_LOG] (Worker5_QueryProcessorQueue10100) END SRC COMMAND:       endTime=2013-01-06 22:39:20.462 requestID=olFjRZvdszew.0 sourceCommandID=3 txID=null modelName=demodata_file      translatorName=demodata_file.delegated-translator sessionID=olFjRZvdszew principal=admin@chorus-login-security finalRowCount=1

06 Jan 2013 22:39:20,462 PST DEBUG [org.teiid.BUFFER_MGR] (Worker17_QueryProcessorQueue10093) Removing TupleBuffer: 8329

06 Jan 2013 22:39:20,462 PST DEBUG [org.teiid.PROCESSOR] (Worker17_QueryProcessorQueue10093) QueryProcessor: closing processor

06 Jan 2013 22:39:20,462 PST DEBUG [org.teiid.BUFFER_MGR] (Worker8_QueryProcessorQueue9904) Blocking on non-final TupleBuffer 8282 size 20: org.teiid.common.buffer.BlockedException

at org.teiid.common.buffer.TupleBuffer$1.finalRow(TupleBuffer.java:329)

at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

at org.teiid.query.processor.relational.SortUtility.incrementWorkingTuple(SortUtility.java:388)

at org.teiid.query.processor.relational.SortUtility.mergePhase(SortUtility.java:337)

at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:188)

at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)

Can you please explain the nature of this exception? Can it indicate a potential resource leak ? If yes any sugestions on how to further diagnose it?

Just in case if it might be related I have also seen org.teiid.common.buffer.impl classes in relation to potential leak suspects in few of our recent OOM heap dumps.

I also have snapshots of CPU usage by method collected over time and can provide it if needed. 

Thank you,

Sabina

[shawkins]

> Can you please explain the nature of this exception?

It indicates the current operation is blocked pending the release of buffer resources, an asynch source query has yet to return, or in this case that the TupleBuffer may not be complete.

> Can it indicate a potential resource leak ?

Generally no.  The buffer space is shared and it is expected to block depending upon memory demands.  In the case of an incomplete TupleBuffer there are only a couple of scenarios where that happens within Teiid.  You'll want to ensure that any relevant issues have been addressed in the version that you are using - for example https://issues.jboss.org/browse/TEIID-2089

> There is a set of 4 or 5 of them for every async execution.

That can be normal depending upon the plan.  The most common issuing of a BlockedException will be If your translators are forkable (returning isForkable() true) and thus having the source query issued asynchronously from the query processing thread.  An attempt to access the results prior to their return will result in a BlockedException.  When not using the calling thread for processing (which is always the case for socket queries) this allows the processing thread to be used for another task.  When using the calling thread (which is typically the case for local / continuous) this may seem somewhat unnecessary, but the functioning is similar in that there may have been other operations issued with-in the plan that can complete even if the current operation has blocked.

> Just in case if it might be related I have also seen org.teiid.common.buffer.impl classes in relation to potential leak suspects in few of our recent OOM heap dumps.

Depending upon you BufferMangager configuration there generally will be a lot of memory held by the BufferManager.  You may need to explicitly tune the BufferManager properties if you have a significant amount of memory usage that is outside of Teiid in the same vm or are using a lot of large object references that Teiid is underestimating the size of.