11 Replies Latest reply on Nov 13, 2015 11:56 AM by shawkins

    Teiid join threading question

    kchen007

      we have an user that tested our server in an old single core machine(windows 7), the VDB had a view that will join from 4 different source tables. when our user run it, he always encountered race condition.

      the VDB use our own resource adapter, which is not thread safe, when I debug it, I found out that Teiid spawn 2 threads to query 2 different tables that is part of the view, but it use the same  resource adapter connection instance.

       

      My understanding is that Teiid will get a different connections from JBOSS' connection pool.

       

      I cannot reproduce this on other workstation with the same workflow.

       

      Is there any way to get around it? also what is the best way to pinpoint where exactly the association is wrong.

       

      thanks

      Kevin

        • 1. Re: Teiid join threading question
          rareddy

          Possibly your resource-adapter's ManagedConnectionFactory and ResourceAdapter class's equals and hashcode method may not have implemented correctly.

          • 2. Re: Teiid join threading question
            shawkins

            > My understanding is that Teiid will get a different connections from JBOSS' connection pool.

             

            Yes, concurrent source queries will use different connections from the pool.  If you have the same connection in use at the same time for concurrent source queries, that is not expected.  Is it possible that the resource adapter is providing the same physical connection multiple times to the connection pooling logic?

             

            > Is there any way to get around it? also what is the best way to pinpoint where exactly the association is wrong.

             

            You can prevent concurrent source access altogether by just executing within the scope of a transaction, which will force the engine to serialize the plan. 

             

            In general if you are getting the same connection concurrently, that's not expected behavior and you'd likely need to have logging at the AS JCA/pooling layer at a debug level or higher and/or debug the resource adapter.

            • 3. Re: Teiid join threading question
              kchen007

              Some more finding, this is happening when we insert a new entity, after that we tried to read the content back to fill auto-generated values.

               

              Our first attempt is to use JPA refresh() to read it back, but the primary key is system generated,  the value is null; so the refresh() will not return anything. When that happened, it appears that Teiid  will return all the connections to connection pool used by the worker thread for the refresh() call or something like that, even the worker thread has not finished; see the 2nd picture, the worker thread are waiting for result, but the connection has been closed and mc=null. when I looked at the actual physical connection from the stacktrace, it is the same as the physical connection used in the 1st picture, another worker thread.

               

              when the refresh() failed, we will try to read the entity back via alternative key columns(which are non-null during insertion), at that time one of the worker will pick up the returned connection(see 1st picture) which is still used in un-returned worker thread from refresh().

               

              this is much easy to reproduce when I associate the process with single processor only.

               

              out_of_seq_thread1.PNG

               

              out_of_seq_thread2.PNG

              • 4. Re: Teiid join threading question
                kchen007

                here are the 2 screen shots that shows one thread is closing the connection that is still running by planner. so, I guess the planner should not close unfinished worker, let them finish and close by themselves.

                this one shows it is closing the connection for worker #21447 from thread Worker57_QPQ3559.

                out_of_seq_close_connection.PNG

                 

                this one shows the worker #21447 is still running

                out_of_seq_cwi_not_finish.PNG

                • 5. Re: Teiid join threading question
                  shawkins

                  I think you are seeing a regression from TEIID-3275.  The fully close source is first canceling the task, which will make it immediately report isDone true, which means that it will immediately run the completion listener.  That is not the intent as we want the closure to happen after the full affect of the cancel.

                  • 6. Re: Teiid join threading question
                    shawkins

                    Actually I may have spoken too soon, while the cancel logic does need adjusting the ConnectorWorkItem is fully synchronized on the execute, more, and close methods.  This means that the close will not happen until after the more call is finished and after that the work item will not be usable.  So I don't think this would explain what you are seeing.

                    • 7. Re: Teiid join threading question
                      shawkins

                      What version are you on?

                      • 8. Re: Teiid join threading question
                        kchen007

                        8.7

                        • 9. Re: Teiid join threading question
                          shawkins

                          This was likely addressed by [TEIID-3612] NPE in connectorworkitem after close - JBoss Issue Tracker as even if close is called when there is still pending work, ConnectoWorkItem.more will not proceed.

                          • 10. Re: Teiid join threading question
                            kchen007

                            that is not the case here. when it happened, the execution is not null. also the connection object(type org.teiid.resource.spi.WrappedConnection) is also not null, but it is state has been changed somewhere else, with close=true and mc=null.

                            • 11. Re: Teiid join threading question
                              shawkins

                              Can you try this with 8.12 or apply the patch to 8.7?