TEIID Services Not available if JBoss AS Running for Couple of Hours.

[akshy_harale]

Hello,

We have an enterprise application running on JBOSS EAP 6.1 with teiid server 8.7. We have 6 data-sources and 3 vdbs We don't have too much load on this server as server is running in development environment.

Our issue:

We kept server running  for couple of ours, after some time we fired some sql query on one of the teiid vdb, but we are getting  exceptions below. And we get different exceptions each time.

1) Caused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])

2) javax.naming.NameNotFoundException: Error looking up teiid/queryengine/embedded, service service jboss.naming.context.java.teiid.queryengine.embedded is not started

3) 2015-08-03 07:18:48,754 WARN  [org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory] (MSC service thread 1-11) Lock owned during cleanup: java.lang.Throwable: Lock owned during cleanup

4)  'TEIID40042 Invalid Session JXmzDu3tFgcD. Session may have already been terminated.' for session null.  Exception type org.teiid.client.security.InvalidSessionException thrown from

What we tried:

1) For blocking time out we tried increasing time for data sources in standalone.xml but that didn't work.

2) For embedded service we followed this post (https://developer.jboss.org/message/935886)

Current workaround for this is to restart the application server.

Please help us to resolve this issue and let us know if you have any input on this.

Thanks,

Akshay

[rareddy]

Akshay,

1) This means that server could not get the connection to the underlying source in a given amount time. Have you checked the source it self, is that is active? Typically idle connections will be flushed after defined time, then new connections are made when requests come in.

2) What operation are you doing when you saw this exception?

3) Not sure about this one.

4) This occurs when client makes a connection, but after that server may have been re-started but the previously made connection is still being used.

We have running servers for days, months together without any issues, looks like there may be something else that may be re-starting your server in between. Have you done any kind of automation with CLI scripts etc to manage the server perhaps? What I would suggest is to leave the server running in DEBUG mode and check the log when you see the symptoms as to what happened to the server prior to that.

Ramesh..

[guydavis]

Hi Ramesh,  I'm following up for Akshay.  We profiled the problematic workflow in Yourkit and found high CPU usage in BufferManagerImpl.doEvictions().  In particular, when the system become unresponsive, this method was consuming 94% of CPU time in this JBoss instance.  Please see below screenshot:

The VDB is a complicated multi-level one connecting to multiple MS SQL Server databases.  Query and procedure results seem to require a substantial amount of buffer space.  Is such CPU usage to be expected in this case?    If this is not expected, what would you suggest we investigate next?  What tuning could be performed to improve BufferManager behavior?

Thanks in advance,

Guy

[shawkins]

More than likely you are seeing [TEIID-3106] BufferManager Cleaner consuming 90% of total CPU time - JBoss Issue Tracker which was addressed for  8.7.1 and 8.9+.

[akshy_harale]

Hello Steven/Ramesh,

We applied the patch on 8.7 for the slow performance issue and we get it resolved. Now we are facing the invalid session issue as mentioned in my question above.

We have a multi-layered VDB architecture where we have Teiid datasource which is connecting to the underlying VDB which is connecting to other datasource like sqlserver.

When I do the connection I get the data but at same time in server.log file I get session invalid exception. I looked into the log file but unable to locate the origin of the session creation.

2015-09-16 06:35:11,255 WARN  [org.teiid.SECURITY] (New I/O worker #7) TEIID40011 Processing exception 'TEIID40042 Invalid Session L1JLlDb3PZU9. Session may have already been terminated.' for session null.  Exception type org.teiid.client.security.InvalidSessionException thrown from org.teiid.services.SessionServiceImpl.getSessionInfo(SessionServiceImpl.java:368).: org.teiid.client.security.InvalidSessionException: TEIID40042 Invalid Session L1JLlDb3PZU9. Session may have already been terminated.

[shawkins]

> When I do the connection I get the data but at same time in server.log file I get session invalid exception. I looked into the log file but unable to locate the origin of the session creation.

Are you seeing unexpected client exceptions?  In some/most circumstances that message is benign.  There were some related cleanups and modifications to the logging of it in later releases, most recently with https://issues.jboss.org/browse/TEIID-3697