12 Replies Latest reply on Mar 29, 2013 10:04 PM by rareddy

    VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...

    sivajd

      Team,

       

      Currently we are using the Teiid 8.3.0.CR1 with local connections and passthrough authentication. We have two vdb's deployed. We  run through our QA TestSuite against each of the VDB's where we are trying to do CRUD operations on all of the entities.

      After running through 55 - 60 of the entities the testsuite stops from running any further because the application server does not respond to any CRUD operations on any of the entities on the first VDB. However the application server responds to basic CRUD operations on the second VDB against which the test suite has not been run.The test suite runs fine against both of the vdb's if we use the socket based connections instead of the local connections and use explicit authentication passing the userid/password.

       

      Why is there a difference in behavior depending on the connection type (local/socket) ? Is there a setting that I am missing out here when I am using local connections.

      The following is the structure of the URL when I am using the local connection : "jdbc:teiid:%s;PassthroughAuthentication=true;fetchSize=2048;ApplicationName=JDBC;VirtualDatabaseName=%s;useCallingThread=false";

       

      We are using the following settings in the teiid-subsystem and thread subsystem when using local connections setup.

       

      Teiid subsystem:

       

              <subsystem xmlns="urn:jboss:domain:teiid:1.0">          

       

                  <async-thread-pool>

                      teiid-async

                  </async-thread-pool>

       

                 <max-threads>

                      150

                  </max-threads>

                  <policy-decider-module>

                      org.jboss.teiid

                  </policy-decider-module>

                  <transport name="embedded">

                      <authentication security-domain="XXXX-Security-Domain"/>

                  </transport>

                

                ...

                ...

                ...

           </subsystem>

       

      Threads subsystem configuration in standalone-teiid.xml

       

             <subsystem xmlns="urn:jboss:domain:threads:1.1">

                  <bounded-queue-thread-pool name="teiid-async">

                      <queue-length count="1000"/>

                      <max-threads count="30"/>

                  </bounded-queue-thread-pool>

              </subsystem>

       

      Any insight into this will be very valuable. Thanks for your help.

       

      regards

      Siva Chemudupaty

        • 1. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
          shawkins

          Has this scenario changed specifically in 8.3 CR1 or was it observed earlier?  This may be somewhat related to https://issues.jboss.org/browse/TEIID-2410 - although the original problem was seeming addressed.  Can you provide a thread dump from when you see the server as unresponsive?  Do you do anything that may cause resource issues such as make connections back to the same Teiid engine from your translators?

           

          > Why is there a difference in behavior depending on the connection type (local/socket) ?

           

          They have a different threading model for one.  socket work is first processed by a socket thread and then handed off to the engine thread pool.  As the work is further processed it will allow the thread to detach as needed when blocked on source operations that have been spawned into other threads.  In the local case the processing thread will remain dedicated to its work and wait for any spawned work to complete.

           

          Steve

          • 2. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
            sivajd

            Steven,

             

            Thanks for your prompt reply.

             

            > Has this scenario changed specifically in 8.3 CR1 or was it observed earlier?  This may be somewhat related to https://issues.jboss.org/browse/TEIID-2410 - although the original problem was seeming addressed.  Can you provide a thread dump from when you see the server as unresponsive?  Do you do anything that may cause resource issues such as make connections back to the same Teiid engine from your translators?

             

            8.2.0.Final version never worked for saving objects using local connections/passthrough authentication / transactions. Kevin Chen and I have worked with Ramesh and you regarding this and you all provided a fix for it in the 8.3 CR1.  Reference link https://community.jboss.org/thread/221016?tstart=0

             

            We are not trying to make any explicit connections back to Teiid engine from within the translators, however we are trying to use a JPA based Cursor to read result sets progressively. The JPA based cursor was created  using the JPA EntityManager and we are trying to reuse the same entity manager when we are navigating the cursor. We also have an eviction mechanism to evict the Cursor after certain inactive time and close the EntityManager too...

             

            I am here by attaching one of the thread dumps from the time when the server hanged and did not respond. Once again thanks for your help.

             

            Thanks

            Siva

            • 3. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
              shawkins

              The thread dump shows no threads actively in Teiid code.  What are you seeing from your connection / query attempts? 

               

              Steve

              • 4. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
                sivajd

                Steve,

                 

                Any request to get data from any entity, time out while the web requests that do not hit the database through teiid are all served up.

                 

                Thanks

                Siva

                • 5. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
                  shawkins

                  You would want to at least see some thread in the Teiid client code (getConnection, etc.) when your attempts are failing for this to directly be a Teiid issue.  You do see 'http-executor-threads - 48' and 'http-executor-threads - 50' waiting to be notified on the same lock from 'org.eclipse.persistence.sessions.server.ServerSession.acquireClientConnection(ServerSession.java:252)'.  That would appear to be some pooling logic in eclipselink that implies connections may not be getting closed as expected.  Can you validate your pooling setting and to see if you have any issues when closing connections.

                   

                  Steve

                  • 6. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
                    shawkins

                    Have you made any progress on this?

                    • 7. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
                      sivajd

                      Steve,

                       

                      I was busy Friday - Monday with some boot camp related work in my project. I started working on this late monday evening and am continuing to look into this.  As far as your question about pooling setting is concerned, we do not explicitly have a pooling setting configured in our eclipselink persistence unit or the datasource that is tied with it. However when I  monitor  the connection pool size from jboss console, I see 20 connections getting created by default from the datasource. When I run the test suite the availability of connections on the datasource keeps changing based on the load.  In the socket based connection mode, the pool size varies from 0% - 92% and runs through the entire test suite, while the pool size gets exhausted and the test suite freezes in the local connections case. In the local connections case I increased the connection pool size to 200. With the connection pool set to 200, the app server uses up 50 connections or so, before the application server freezes. It still has 150 connections but the threads are all getting locked up as shown in the thread dump. During the partial run of the test suite there are some successful tests and some failed tests. I am examining the transaction related code to check if we are closing the entity  manager appropriately. Based on my initial examination, I find everything to be as per my expectation.

                       

                      Should you have any suggestions, please do let me know and I would be more than glad to try them out...

                       

                      Thanks

                      Siva

                      • 8. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
                        shawkins

                        >   As far as your question about pooling setting is concerned, we do not explicitly have a pooling setting configured in our eclipselink persistence unit or the datasource that is tied with it.

                         

                        From a quick look at their source it looked like that would be related to its internal notion of a connection pool.

                         

                        > However when I  monitor  the connection pool size from jboss console, I see 20 connections getting created by default from the datasource.

                         

                        Do you mean a datasource created for TeiidDataSource or a source that is accessed below Teiid?

                         

                        > Should you have any suggestions, please do let me know and I would be more than glad to try them out...

                         

                        It seems like you have to know what the purpose of the eclipselink lock is and why connections are just waiting.  You can get the source for the version of eclipselink you are running and see if the code matches your expectations or step through a debug.

                         

                        Steve

                        • 9. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
                          sivajd

                          Steve,

                           

                          As per your suggestion, debugged the eclipselink api and changed the api/settings on our side to overcome the issue we were facing at the eclipselink api level. Effectively eclipselink  has a default limit of 50 for the NonPooled connections at the JPA level, Once we changed it based on our requirements we are all able to run through the QA  testing.

                          Now we are running into a new issue where the datasource based connections that we are creating/using at the Teiid level are not getting released back to the datasource pool. Effectively we are seeing leakage of connections.  Allocation of the connections are happening as shown below in the stack trace

                          and these are not getting released back to the datasource pool. Any suggestions or help regarding this would be very helpful.

                           

                          com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(String, Properties)

                          org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection()

                          org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(DataSource)

                          com.lgc.edm.teiid.translator.EdmSqlServerExecutionFactory.getConnection(DataSource)

                          com.lgc.edm.teiid.translator.EdmSqlServerExecutionFactory.getConnection(Object)

                          org.teiid.translator.ExecutionFactory.getConnection(Object, ExecutionContext)

                          org.teiid.translator.BaseDelegatingExecutionFactory.getConnection(Object, ExecutionContext)

                          org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute()

                          org.teiid.dqp.internal.process.DataTierTupleSource.getResults()

                          org.teiid.dqp.internal.process.DataTierTupleSource$1.call()<2 recursive calls>

                          java.util.concurrent.FutureTask.run()

                          org.teiid.dqp.internal.process.DQPCore$FutureWork.run()

                          org.teiid.dqp.internal.process.DQPWorkContext.runInContext(Runnable)

                          org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run()

                          org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run()

                          java.lang.Thread.run()

                           

                           

                           

                          Thanks

                          Siva

                          • 10. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
                            shawkins

                            That is a normal stack for opening a connection.  You would also want to see ExecutionFactory.closeConnection being called when the Execution is closed.  If using a custom translator, then make sure the closeConnection method is implemented correctly as well.

                             

                            Steve

                            • 11. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
                              sivajd

                              Steve,

                               

                              We have  implemented the closeConnection method in our translator. All we do is simply invoke the super.closeConnection() in the implementation.

                               

                              When does the ExecutionFactory.closeConnection(Connection, DataSource ) get called. I tried to debug through the process and I do not see this method ever getting invoked.

                               

                              Thanks

                              Siva

                              • 12. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
                                rareddy

                                When the query is finished, and results are fetched on translator, close will be called. If u set the "keepExecutionAlive" to true on ExecutionContext, then it will not be closed until the client query is closed.