1 2 Previous Next 16 Replies Latest reply on Jan 20, 2015 9:26 AM by shawkins

    Queries getting blocked after the message "reached max source concurrency of 6"

    madhu.garimilla

      I am using EAP6.1+Teiid8.7. My application has a union query that has some materialized views in it. While this query is getting executed , it is getting blocked after the message "reached max source concurrency of  6" . Attached the log output for this case. I could not able to figure out what's going wrong here. Could someone please check and let us know

        • 1. Re: Queries getting blocked after the message "reached max source concurrency of 6"
          shawkins

          To prevent a single user query from using too many connections / resources, the engine by default will limit the number of source queries it can issue at a time.  The default is determined based upon max engine threads and max active plans - which is 6 for those default values.  Increasing the max threads, decreasing the max active plans, or directly setting a higher value for user request source concurrency will allow more concurrent source queries from a single user query.

          • 2. Re: Queries getting blocked after the message "reached max source concurrency of 6"
            madhu.garimilla

            Steve,

             

            Our Application has the below setting in standalone.

            <max-threads>256</max-threads>

            <max-active-plans>80</max-active-plans>

             

            for which the calculated max source concurrency was 6 because of which queries are getting blocked.

            So,I tried with various numbers and finally i got it working by setting it to 17 in standalone like this.

             

            <thread-count-for-source-concurrency>17</thread-count-for-source-concurrency>

             

            I would like to know how this setting is going to impact my application. Could you please provide some inputs on this.

            • 3. Re: Queries getting blocked after the message "reached max source concurrency of 6"
              shawkins

              > I would like to know how this setting is going to impact my application. Could you please provide some inputs on this.

               

              It means that each user query will be allowed to spawn at most 17 concurrent source requests.  With max active plans at 80, that could be as many as 1360 concurrent source queries.  As long as your connection pools and backend system can handle that load, then there is no problem - otherwise you could just end up moving the point where there is contention - such as seeing timeouts getting a connection from a pool.

              • 4. Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
                madhu.garimilla

                Steve, after increasing the concurrent source requests it is no more blocking on it. However, it gets blocked randomly on some view which is part of the original union query. Attached the log for this request.

                 

                23 Dec 2014 18:32:16,611 INFO  [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-1) kHvXACZPMZFa TEIID30013 Loading materialized view table #MAT_GSV.SYSTEMS_VIEW

                23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Creating STree: 5383

                23 Dec 2014 18:32:16,611 DEBUG [org.teiid.CONNECTOR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa kHvXACZPMZFa.47.159.1567 Create State

                23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Creating TupleBuffer: 5385 [a__9.Sysplex, a__9."System"] [class java.lang.String, class java.lang.String] batch size 512 of type PROCESSOR

                23 Dec 2014 18:32:16,611 DEBUG [org.teiid.PROCESSOR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Created intermediate sort buffer 5385

                23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa kHvXACZPMZFa.47.159.1567 Blocking on source query kHvXACZPMZFa.47.159.1567

                23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa kHvXACZPMZFa.47 Blocking on source request(s).

                23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Removing TupleBuffer: 5385

                23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Creating TupleBuffer: 5386 [a__9.Sysplex, a__9."System"] [class java.lang.String, class java.lang.String] batch size 512 of type PROCESSOR

                23 Dec 2014 18:32:16,611 DEBUG [org.teiid.PROCESSOR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Created intermediate sort buffer 5386

                23 Dec 2014 18:32:16,611 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Blocking on non-final TupleBuffer 5386 size 0: org.teiid.common.buffer.BlockedException

                    at org.teiid.common.buffer.TupleBuffer$TupleBufferTupleSource.finalRow(TupleBuffer.java:61)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortNode.outputPhase(SortNode.java:133)

                    at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:94)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:278)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:136)

                 

                23 Dec 2014 18:32:16,621 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa kHvXACZPMZFa.47 Blocking on source request(s).

                23 Dec 2014 18:32:16,621 DEBUG [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-1) kHvXACZPMZFa org.teiid.query.tempdata.GlobalTableStoreImpl$MatTableInfo@497e83d7 invalidating

                23 Dec 2014 18:32:16,631 DEBUG [org.teiid.PROCESSOR] (Worker66_QueryProcessorQueue3067) kHvXACZPMZFa Running task for parent thread http-/0.0.0.0:8080-1

                23 Dec 2014 18:32:16,631 DEBUG [org.teiid.PROCESSOR] (Worker62_QueryProcessorQueue3066) kHvXACZPMZFa Running task for parent thread http-/0.0.0.0:8080-1

                23 Dec 2014 18:32:16,631 DEBUG [org.teiid.CONNECTOR] (Worker66_QueryProcessorQueue3067) kHvXACZPMZFa kHvXACZPMZFa.47.159.1567 Processing NEW request: SELECT GSV.SYSTEMS.Sysplex, GSV.SYSTEMS."System" FROM GSV.SYSTEMS

                23 Dec 2014 18:32:16,631 DEBUG [org.teiid.CONNECTOR] (Worker62_QueryProcessorQueue3066) kHvXACZPMZFa kHvXACZPMZFa.47.152.1563 Processing NEW request: SELECT performance_netmaster.SDFROM.Sysplex, performance_netmaster.SDFROM."System" FROM performance_netmaster.SDFROM WHERE (performance_netmaster.SDFROM.KEY_DISTFROMADDR <> '-') AND (performance_netmaster.SDFROM.KEY_LOCALPORT = 0)

                23 Dec 2014 18:32:16,631 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa Blocking on mat view load #MAT_GSV.SYSTEMS_VIEW

                23 Dec 2014 18:32:16,631 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa kHvXACZPMZFa.47 Blocking on source request(s).

                23 Dec 2014 18:32:16,631 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-1) kHvXACZPMZFa kHvXACZPMZFa.47 Blocking on union source. 28

                • 5. Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
                  shawkins

                  It should be that the mat view load will resume after the blockedexception, but instead it looks like the logic thinks the load is still on-going and waits.  Would it be possible for you to retry your scenario in a later release to see if this has already been addressed?

                  • 6. Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
                    madhu.garimilla

                    Steve, what is the latest teiid version supported for EAP 6.1? I tried with teiid 8.9 CR1 but there were lot of exceptions and my jboss didn't start.

                    • 7. Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
                      shawkins

                      Yes 8.9 CR1 should be the last for 6.1.  Later would use EAP 6.3 Alpha.

                      • 8. Re: Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
                        madhu.garimilla

                        Steve, I am able to test with teiid 8.9 CR1 and i still see the same problem. There is no exception in the logs but the query gets blocked on loading of some materialized views. I have observed the same problem in a different use case in our application where it gets blocked on the materialized views loading. Could you please check once. attached the log snippet.

                         

                         

                        30 Dec 2014 17:53:56,831 DEBUG [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-3) bLHG5cj67fRq org.teiid.query.tempdata.GlobalTableStoreImpl$MatTableInfo@2d7ff008 invalidating

                        30 Dec 2014 17:53:56,831 DEBUG [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-3) bLHG5cj67fRq org.teiid.query.tempdata.GlobalTableStoreImpl$MatTableInfo@2d7ff008 setting matState to NEEDS_LOADING null 1419942236831 old values LOADED false

                        30 Dec 2014 17:53:56,831 DEBUG [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-3) bLHG5cj67fRq org.teiid.query.tempdata.GlobalTableStoreImpl$MatTableInfo@2d7ff008 invalidating

                        30 Dec 2014 17:53:56,831 DEBUG [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-3) bLHG5cj67fRq org.teiid.query.tempdata.GlobalTableStoreImpl$MatTableInfo@2d7ff008 setting matState to LOADING null 1419942236831 old values NEEDS_LOADING false

                        30 Dec 2014 17:53:56,831 INFO  [org.teiid.PROCESSOR.MATVIEWS] (http-/0.0.0.0:8080-3) bLHG5cj67fRq TEIID30013 Loading materialized view table #MAT_GSV.ALERTS_VIEW

                        30 Dec 2014 17:53:56,831 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Creating STree: 1497

                        30 Dec 2014 17:53:56,837 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Creating TupleBuffer: 1499 [SYSTEMS_VIEW__9.Sysplex, SYSTEMS_VIEW__9."System"] [class java.lang.String, class java.lang.String] batch size 512 of type PROCESSOR

                        30 Dec 2014 17:53:56,837 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Creating TupleBuffer: 1500 [#MAT_GSV.SYSTEMS_VIEW.Sysplex, #MAT_GSV.SYSTEMS_VIEW."System"] [class java.lang.String, class java.lang.String] batch size 512 of type PROCESSOR

                        30 Dec 2014 17:53:56,837 DEBUG [org.teiid.PROCESSOR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Created intermediate sort buffer 1500

                        30 Dec 2014 17:53:56,837 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Creating TupleBuffer: 1501 [#MAT_GSV.SYSTEMS_VIEW.Sysplex, #MAT_GSV.SYSTEMS_VIEW."System"] [class java.lang.String, class java.lang.String] batch size 512 of type PROCESSOR

                        30 Dec 2014 17:53:56,837 DEBUG [org.teiid.PROCESSOR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Created intermediate sort buffer 1501

                        30 Dec 2014 17:53:56,837 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Removing TupleBuffer: 1500

                        30 Dec 2014 17:53:56,837 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Removing TupleBuffer: 1501

                        30 Dec 2014 17:53:56,837 DEBUG [org.teiid.CONNECTOR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq bLHG5cj67fRq.52.162.409 Create State

                        30 Dec 2014 17:53:56,838 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq Creating STree: 1502

                        30 Dec 2014 17:53:56,838 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq bLHG5cj67fRq.52.162.409 Blocking on source query bLHG5cj67fRq.52.162.409

                        30 Dec 2014 17:53:56,838 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq bLHG5cj67fRq.52 Blocking on source request(s).

                        30 Dec 2014 17:53:56,838 DEBUG [org.teiid.PROCESSOR] (Worker28_QueryProcessorQueue1212) bLHG5cj67fRq Running task for parent thread http-/0.0.0.0:8080-3

                        30 Dec 2014 17:53:56,838 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq bLHG5cj67fRq.52 Blocking on source request(s).

                        30 Dec 2014 17:53:56,838 DEBUG [org.teiid.BUFFER_MGR] (http-/0.0.0.0:8080-3) bLHG5cj67fRq bLHG5cj67fRq.52 Blocking on union source. 26

                        • 9. Re: Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
                          madhu.garimilla

                          Steve, Shall i report a JIRA for this as i see the problem exists with latest Teiid (8.9 CR1) also ?

                          • 10. Re: Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
                            shawkins

                            Yes, please do.

                            • 12. Re: Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
                              shawkins

                              The log from #8 does not seem to show any inappropriate behavior.  All 4 materialized views involved load successfully and it doesn't show anything at the end except that it is waiting on some source queries.

                              • 13. Re: Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
                                shawkins

                                Can you follow up on my last comment.  Is there an 8.9+ log that shows inappropriate behavior?

                                • 14. Re: Re: Re: Queries getting blocked after the message "reached max source concurrency of 6"
                                  madhu.garimilla

                                  Hi Steve,

                                   

                                  I did not see anything inappropriate from 8.9+ logs. I see that materialized views are loaded but i couldn't figure out anything about the source queries which were getting blocked.

                                  1 2 Previous Next