-
1. Re: Queries getting blocked after the message "reached max source concurrency of 6"
shawkins Dec 18, 2014 8:06 AM (in response to madhu.garimilla)To prevent a single user query from using too many connections / resources, the engine by default will limit the number of source queries it can issue at a time. The default is determined based upon max engine threads and max active plans - which is 6 for those default values. Increasing the max threads, decreasing the max active plans, or directly setting a higher value for user request source concurrency will allow more concurrent source queries from a single user query.
-
2. Re: Queries getting blocked after the message "reached max source concurrency of 6"
madhu.garimilla Dec 22, 2014 5:32 AM (in response to shawkins)Steve,
Our Application has the below setting in standalone.
<max-threads>256</max-threads>
<max-active-plans>80</max-active-plans>
for which the calculated max source concurrency was 6 because of which queries are getting blocked.
So,I tried with various numbers and finally i got it working by setting it to 17 in standalone like this.
<thread-count-for-source-concurrency>17</thread-count-for-source-concurrency>
I would like to know how this setting is going to impact my application. Could you please provide some inputs on this.
-
3. Re: Queries getting blocked after the message "reached max source concurrency of 6"
shawkins Dec 22, 2014 8:07 AM (in response to madhu.garimilla)> I would like to know how this setting is going to impact my application. Could you please provide some inputs on this.
It means that each user query will be allowed to spawn at most 17 concurrent source requests. With max active plans at 80, that could be as many as 1360 concurrent source queries. As long as your connection pools and backend system can handle that load, then there is no problem - otherwise you could just end up moving the point where there is contention - such as seeing timeouts getting a connection from a pool.
-
4. Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
madhu.garimilla Dec 23, 2014 8:20 AM (in response to shawkins)Steve, after increasing the concurrent source requests it is no more blocking on it. However, it gets blocked randomly on some view which is part of the original union query. Attached the log for this request.
23 Dec 2014 18:32:16,611 INFO [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-1) kHvXACZPMZFa TEIID30013 Loading materialized view table #MAT_GSV.SYSTEMS_VIEW
23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Creating STree: 5383
23 Dec 2014 18:32:16,611 DEBUG [org.teiid.CONNECTOR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa kHvXACZPMZFa.47.159.1567 Create State
23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Creating TupleBuffer: 5385 [a__9.Sysplex, a__9."System"] [class java.lang.String, class java.lang.String] batch size 512 of type PROCESSOR
23 Dec 2014 18:32:16,611 DEBUG [org.teiid.PROCESSOR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Created intermediate sort buffer 5385
23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa kHvXACZPMZFa.47.159.1567 Blocking on source query kHvXACZPMZFa.47.159.1567
23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa kHvXACZPMZFa.47 Blocking on source request(s).
23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Removing TupleBuffer: 5385
23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Creating TupleBuffer: 5386 [a__9.Sysplex, a__9."System"] [class java.lang.String, class java.lang.String] batch size 512 of type PROCESSOR
23 Dec 2014 18:32:16,611 DEBUG [org.teiid.PROCESSOR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Created intermediate sort buffer 5386
23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Blocking on non-final TupleBuffer 5386 size 0: org.teiid.common.buffer.BlockedException
at org.teiid.common.buffer.TupleBuffer$TupleBufferTupleSource.finalRow(TupleBuffer.java:61)
at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:70)
at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)
at org.teiid.query.processor.relational.SortNode.outputPhase(SortNode.java:133)
at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:94)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:278)
at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:136)
23 Dec 2014 18:32:16,621 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa kHvXACZPMZFa.47 Blocking on source request(s).
23 Dec 2014 18:32:16,621 DEBUG [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-1) kHvXACZPMZFa org.teiid.query.tempdata.GlobalTableStoreImpl$MatTableInfo@497e83d7 invalidating
23 Dec 2014 18:32:16,631 DEBUG [org.teiid.PROCESSOR] (Worker66_QueryProcessorQueue3067) kHvXACZPMZFa Running task for parent thread http-/0.0.0.0:8080-1
23 Dec 2014 18:32:16,631 DEBUG [org.teiid.PROCESSOR] (Worker62_QueryProcessorQueue3066) kHvXACZPMZFa Running task for parent thread http-/0.0.0.0:8080-1
23 Dec 2014 18:32:16,631 DEBUG [org.teiid.CONNECTOR] (Worker66_QueryProcessorQueue3067) kHvXACZPMZFa kHvXACZPMZFa.47.159.1567 Processing NEW request: SELECT GSV.SYSTEMS.Sysplex, GSV.SYSTEMS."System" FROM GSV.SYSTEMS
23 Dec 2014 18:32:16,631 DEBUG [org.teiid.CONNECTOR] (Worker62_QueryProcessorQueue3066) kHvXACZPMZFa kHvXACZPMZFa.47.152.1563 Processing NEW request: SELECT performance_netmaster.SDFROM.Sysplex, performance_netmaster.SDFROM."System" FROM performance_netmaster.SDFROM WHERE (performance_netmaster.SDFROM.KEY_DISTFROMADDR <> '-') AND (performance_netmaster.SDFROM.KEY_LOCALPORT = 0)
23 Dec 2014 18:32:16,631 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Blocking on mat view load #MAT_GSV.SYSTEMS_VIEW
23 Dec 2014 18:32:16,631 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa kHvXACZPMZFa.47 Blocking on source request(s).
23 Dec 2014 18:32:16,631 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa kHvXACZPMZFa.47 Blocking on union source. 28
-
server.log.zip 9.2 KB
-
-
5. Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
shawkins Dec 23, 2014 11:09 AM (in response to madhu.garimilla)It should be that the mat view load will resume after the blockedexception, but instead it looks like the logic thinks the load is still on-going and waits. Would it be possible for you to retry your scenario in a later release to see if this has already been addressed?
-
6. Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
madhu.garimilla Dec 29, 2014 7:16 AM (in response to shawkins)Steve, what is the latest teiid version supported for EAP 6.1? I tried with teiid 8.9 CR1 but there were lot of exceptions and my jboss didn't start.
-
7. Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
shawkins Dec 29, 2014 9:10 AM (in response to madhu.garimilla)Yes 8.9 CR1 should be the last for 6.1. Later would use EAP 6.3 Alpha.
-
8. Re: Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
madhu.garimilla Dec 30, 2014 7:52 AM (in response to shawkins)Steve, I am able to test with teiid 8.9 CR1 and i still see the same problem. There is no exception in the logs but the query gets blocked on loading of some materialized views. I have observed the same problem in a different use case in our application where it gets blocked on the materialized views loading. Could you please check once. attached the log snippet.
30 Dec 2014 17:53:56,831 DEBUG [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-3) bLHG5cj67fRq org.teiid.query.tempdata.GlobalTableStoreImpl$MatTableInfo@2d7ff008 invalidating
30 Dec 2014 17:53:56,831 DEBUG [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-3) bLHG5cj67fRq org.teiid.query.tempdata.GlobalTableStoreImpl$MatTableInfo@2d7ff008 setting matState to NEEDS_LOADING null 1419942236831 old values LOADED false
30 Dec 2014 17:53:56,831 DEBUG [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-3) bLHG5cj67fRq org.teiid.query.tempdata.GlobalTableStoreImpl$MatTableInfo@2d7ff008 invalidating
30 Dec 2014 17:53:56,831 DEBUG [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-3) bLHG5cj67fRq org.teiid.query.tempdata.GlobalTableStoreImpl$MatTableInfo@2d7ff008 setting matState to LOADING null 1419942236831 old values NEEDS_LOADING false
30 Dec 2014 17:53:56,831 INFO [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-3) bLHG5cj67fRq TEIID30013 Loading materialized view table #MAT_GSV.ALERTS_VIEW
30 Dec 2014 17:53:56,831 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Creating STree: 1497
30 Dec 2014 17:53:56,837 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Creating TupleBuffer: 1499 [SYSTEMS_VIEW__9.Sysplex, SYSTEMS_VIEW__9."System"] [class java.lang.String, class java.lang.String] batch size 512 of type PROCESSOR
30 Dec 2014 17:53:56,837 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Creating TupleBuffer: 1500 [#MAT_GSV.SYSTEMS_VIEW.Sysplex, #MAT_GSV.SYSTEMS_VIEW."System"] [class java.lang.String, class java.lang.String] batch size 512 of type PROCESSOR
30 Dec 2014 17:53:56,837 DEBUG [org.teiid.PROCESSOR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Created intermediate sort buffer 1500
30 Dec 2014 17:53:56,837 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Creating TupleBuffer: 1501 [#MAT_GSV.SYSTEMS_VIEW.Sysplex, #MAT_GSV.SYSTEMS_VIEW."System"] [class java.lang.String, class java.lang.String] batch size 512 of type PROCESSOR
30 Dec 2014 17:53:56,837 DEBUG [org.teiid.PROCESSOR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Created intermediate sort buffer 1501
30 Dec 2014 17:53:56,837 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Removing TupleBuffer: 1500
30 Dec 2014 17:53:56,837 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Removing TupleBuffer: 1501
30 Dec 2014 17:53:56,837 DEBUG [org.teiid.CONNECTOR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq bLHG5cj67fRq.52.162.409 Create State
30 Dec 2014 17:53:56,838 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Creating STree: 1502
30 Dec 2014 17:53:56,838 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq bLHG5cj67fRq.52.162.409 Blocking on source query bLHG5cj67fRq.52.162.409
30 Dec 2014 17:53:56,838 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq bLHG5cj67fRq.52 Blocking on source request(s).
30 Dec 2014 17:53:56,838 DEBUG [org.teiid.PROCESSOR] (Worker28_QueryProcessorQueue1212) bLHG5cj67fRq Running task for parent thread http-/0.0.0.0:8080-3
30 Dec 2014 17:53:56,838 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq bLHG5cj67fRq.52 Blocking on source request(s).
30 Dec 2014 17:53:56,838 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq bLHG5cj67fRq.52 Blocking on union source. 26
-
treeloading_server.log.zip 21.4 KB
-
-
9. Re: Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
madhu.garimilla Jan 5, 2015 1:54 AM (in response to shawkins)Steve, Shall i report a JIRA for this as i see the problem exists with latest Teiid (8.9 CR1) also ?
-
10. Re: Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
shawkins Jan 5, 2015 1:37 PM (in response to madhu.garimilla)Yes, please do.
-
12. Re: Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
shawkins Jan 6, 2015 1:15 PM (in response to madhu.garimilla)The log from #8 does not seem to show any inappropriate behavior. All 4 materialized views involved load successfully and it doesn't show anything at the end except that it is waiting on some source queries.
-
13. Re: Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
shawkins Jan 14, 2015 4:03 PM (in response to shawkins)Can you follow up on my last comment. Is there an 8.9+ log that shows inappropriate behavior?
-
14. Re: Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
madhu.garimilla Jan 16, 2015 7:40 AM (in response to shawkins)Hi Steve,
I did not see anything inappropriate from 8.9+ logs. I see that materialized views are loaded but i couldn't figure out anything about the source queries which were getting blocked.