-
1. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
shawkins Mar 7, 2013 6:38 PM (in response to sivajd)Has this scenario changed specifically in 8.3 CR1 or was it observed earlier? This may be somewhat related to https://issues.jboss.org/browse/TEIID-2410 - although the original problem was seeming addressed. Can you provide a thread dump from when you see the server as unresponsive? Do you do anything that may cause resource issues such as make connections back to the same Teiid engine from your translators?
> Why is there a difference in behavior depending on the connection type (local/socket) ?
They have a different threading model for one. socket work is first processed by a socket thread and then handed off to the engine thread pool. As the work is further processed it will allow the thread to detach as needed when blocked on source operations that have been spawned into other threads. In the local case the processing thread will remain dedicated to its work and wait for any spawned work to complete.
Steve
-
2. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
sivajd Mar 7, 2013 8:13 PM (in response to shawkins)Steven,
Thanks for your prompt reply.
> Has this scenario changed specifically in 8.3 CR1 or was it observed earlier? This may be somewhat related to https://issues.jboss.org/browse/TEIID-2410 - although the original problem was seeming addressed. Can you provide a thread dump from when you see the server as unresponsive? Do you do anything that may cause resource issues such as make connections back to the same Teiid engine from your translators?
8.2.0.Final version never worked for saving objects using local connections/passthrough authentication / transactions. Kevin Chen and I have worked with Ramesh and you regarding this and you all provided a fix for it in the 8.3 CR1. Reference link https://community.jboss.org/thread/221016?tstart=0
We are not trying to make any explicit connections back to Teiid engine from within the translators, however we are trying to use a JPA based Cursor to read result sets progressively. The JPA based cursor was created using the JPA EntityManager and we are trying to reuse the same entity manager when we are navigating the cursor. We also have an eviction mechanism to evict the Cursor after certain inactive time and close the EntityManager too...
I am here by attaching one of the thread dumps from the time when the server hanged and did not respond. Once again thanks for your help.
Thanks
Siva
-
as7-td3.log.zip 7.4 KB
-
-
3. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
shawkins Mar 7, 2013 11:05 PM (in response to sivajd)The thread dump shows no threads actively in Teiid code. What are you seeing from your connection / query attempts?
Steve
-
4. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
sivajd Mar 8, 2013 9:05 AM (in response to shawkins)Steve,
Any request to get data from any entity, time out while the web requests that do not hit the database through teiid are all served up.
Thanks
Siva
-
5. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
shawkins Mar 8, 2013 9:19 AM (in response to sivajd)You would want to at least see some thread in the Teiid client code (getConnection, etc.) when your attempts are failing for this to directly be a Teiid issue. You do see 'http-executor-threads - 48' and 'http-executor-threads - 50' waiting to be notified on the same lock from 'org.eclipse.persistence.sessions.server.ServerSession.acquireClientConnection(ServerSession.java:252)'. That would appear to be some pooling logic in eclipselink that implies connections may not be getting closed as expected. Can you validate your pooling setting and to see if you have any issues when closing connections.
Steve
-
6. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
shawkins Mar 12, 2013 10:06 AM (in response to shawkins)Have you made any progress on this?
-
7. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
sivajd Mar 12, 2013 4:58 PM (in response to shawkins)Steve,
I was busy Friday - Monday with some boot camp related work in my project. I started working on this late monday evening and am continuing to look into this. As far as your question about pooling setting is concerned, we do not explicitly have a pooling setting configured in our eclipselink persistence unit or the datasource that is tied with it. However when I monitor the connection pool size from jboss console, I see 20 connections getting created by default from the datasource. When I run the test suite the availability of connections on the datasource keeps changing based on the load. In the socket based connection mode, the pool size varies from 0% - 92% and runs through the entire test suite, while the pool size gets exhausted and the test suite freezes in the local connections case. In the local connections case I increased the connection pool size to 200. With the connection pool set to 200, the app server uses up 50 connections or so, before the application server freezes. It still has 150 connections but the threads are all getting locked up as shown in the thread dump. During the partial run of the test suite there are some successful tests and some failed tests. I am examining the transaction related code to check if we are closing the entity manager appropriately. Based on my initial examination, I find everything to be as per my expectation.
Should you have any suggestions, please do let me know and I would be more than glad to try them out...
Thanks
Siva
-
8. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
shawkins Mar 14, 2013 9:02 AM (in response to sivajd)> As far as your question about pooling setting is concerned, we do not explicitly have a pooling setting configured in our eclipselink persistence unit or the datasource that is tied with it.
From a quick look at their source it looked like that would be related to its internal notion of a connection pool.
> However when I monitor the connection pool size from jboss console, I see 20 connections getting created by default from the datasource.
Do you mean a datasource created for TeiidDataSource or a source that is accessed below Teiid?
> Should you have any suggestions, please do let me know and I would be more than glad to try them out...
It seems like you have to know what the purpose of the eclipselink lock is and why connections are just waiting. You can get the source for the version of eclipselink you are running and see if the code matches your expectations or step through a debug.
Steve
-
9. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
sivajd Mar 29, 2013 4:51 PM (in response to shawkins)Steve,
As per your suggestion, debugged the eclipselink api and changed the api/settings on our side to overcome the issue we were facing at the eclipselink api level. Effectively eclipselink has a default limit of 50 for the NonPooled connections at the JPA level, Once we changed it based on our requirements we are all able to run through the QA testing.
Now we are running into a new issue where the datasource based connections that we are creating/using at the Teiid level are not getting released back to the datasource pool. Effectively we are seeing leakage of connections. Allocation of the connections are happening as shown below in the stack trace
and these are not getting released back to the datasource pool. Any suggestions or help regarding this would be very helpful.
com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(String, Properties)
org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection()
org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(DataSource)
com.lgc.edm.teiid.translator.EdmSqlServerExecutionFactory.getConnection(DataSource)
com.lgc.edm.teiid.translator.EdmSqlServerExecutionFactory.getConnection(Object)
org.teiid.translator.ExecutionFactory.getConnection(Object, ExecutionContext)
org.teiid.translator.BaseDelegatingExecutionFactory.getConnection(Object, ExecutionContext)
org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute()
org.teiid.dqp.internal.process.DataTierTupleSource.getResults()
org.teiid.dqp.internal.process.DataTierTupleSource$1.call()<2 recursive calls>
java.util.concurrent.FutureTask.run()
org.teiid.dqp.internal.process.DQPCore$FutureWork.run()
org.teiid.dqp.internal.process.DQPWorkContext.runInContext(Runnable)
org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run()
org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run()
java.lang.Thread.run()
Thanks
Siva
-
10. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
shawkins Mar 29, 2013 6:06 PM (in response to sivajd)That is a normal stack for opening a connection. You would also want to see ExecutionFactory.closeConnection being called when the Execution is closed. If using a custom translator, then make sure the closeConnection method is implemented correctly as well.
Steve
-
11. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
sivajd Mar 29, 2013 7:06 PM (in response to shawkins)Steve,
We have implemented the closeConnection method in our translator. All we do is simply invoke the super.closeConnection() in the implementation.
When does the ExecutionFactory.closeConnection(Connection, DataSource ) get called. I tried to debug through the process and I do not see this method ever getting invoked.
Thanks
Siva
-
12. Re: VDB deployed with Local Connections and Passthrough Authentication Freezes against our QA TestSuite...
rareddy Mar 29, 2013 10:04 PM (in response to sivajd)When the query is finished, and results are fetched on translator, close will be called. If u set the "keepExecutionAlive" to true on ExecutionContext, then it will not be closed until the client query is closed.