Server recovery after "No Managed Connections Avaiable" exce

[aciccare]

Our JBoss servers are crashing periodically. The server logs show multiple iterations of NestedSQLException: No Managed Connections Available. Is there a way to keep the servers from crashing and have them wait for a managed connection to become available?

[dcartier]

I just experienced this on one of my node this weekend. Only happened on 1 out of 4 machines though.

I am not sure how long it went on for, it was the middle of the night, but the server seemed to recover on its own.

The servers have pretty much zero traffic during this period, so I do not think it was load related.

Dennis

[jmboone]

I too have had problems on occasion with the pool manager running out of connections. The only thing I can link it to is database maintenance on Sunday. The DBs (DB2) are shutdown for a cold backup. When they come back, JBoss may or may not connect to the database properly. The pool will throw the No Managed Connections exception even when the DB is up and running fine. We have to touch the service.xml files to cause the pool to reinitialize and hand out connections again.

I feel this is a bug somewhere, but I an not certain how to proceed. There is virtually no way I know of to look inside the pools during operation. Instrumentation is lacking. Configuration options are also limited. The pool managers are effectively black boxes so who knows what is going on?

We have had good luck with Poolman in the past. Under the same load, Poolman retained one DB connection open to service many requests. The native JBoss pool managers seem to open many more connections (like 15) to manage the same load. I am tempted to integrate Poolman with JBoss and bypass the native DB connection pools unless someone can convince me otherwise!

Oh, using JBoss 3.0.4 with Tomcat 4.1.12 running on AIX connecting to DB2 databases (7 of them)

[thl-mot]

We have the same/very similar problem.

Environment:
JBoss Release: JBoss-3.0.4 CVSTag=JBoss_3_0_4
Tomcat: 4.1
Database: MS SQL-Server 2000
Entities are all CMP 2.0


In our project we have discovered several server crashes during the last 2 month.
At this moment we see lots of Exceptions telling "No Managed Connections available".
This sometimes happens after 5-15 Days of up time, but we also had this behaviour multible times a day.

Here is a part of the DB-Connector XML file:

<depends optional-attribute-name="ManagedConnectionPool">
<!--embedded mbean-->


0
50
5000
15
ByContainer




In normal operation (according to the jmx-comsole) we usually have about 2-5 used connections with about 300 concurrent users on the web-site.
If we have a look at the database when we get the exceptions we can't see any locks.
I once had watched a situation, where the database cpu usage was 100%: A second application (Cold-Fusion) is using the same DB-Server. At the same time the available connections (jmx-console) went down.
I think that Jboss is starting to open connections, as dozens of users ar using the application, nearly every click starts a new connection. When reaching the maximum number of connections available for the pool, the exceptions will be thrown.
When CPU usage of the db-server is going down to normal usage, all open connections should end there work, and be released from the connection-pool. But the last time the pool still kept all connections used.
Shouldn't jboss recover from the situation?

Why is jboss keaping an idle connection in the used pool. To me this seams to be an error in the db-pool manager.

I accept that the connections to the database aren't closed (thats the sense of a pool), but if the (jboss-) transaction is ready it should immediately be reuseable for the next thread.

The parameter for "IdleTimeoutMinutes" will handle the close of unused physical db-connection. If jboss can't reuse an already open connection, I think, it won't be closed after this timeout, because it's still used by a jboss thread even if no traffic is going on, right? At least the jmx-console is telling me thaat.
What can I do now, to make this application run 24/7 without?

[jgoodling]

Did anyone ever find a fix for this problem? We are having a very similiar problem.

[ericmolitor]

I just had the joy of walking through the JCA code and the JDBC Wrapper code in regards to similar problems. In every case I checked its been connection leaks in the application (getting a connection from the pool but not calling close to return it to the pool.) The only scary part of this right now (from what we can tell) is that the timeouts are never reaping connections. Create a simple app that grabs a connection and never returns it, set your timeouts for 15 minutes and come back in a few hours and look at the jmx-console for the connection pool and you will see the connection was never reaped.

I'm starting a thread in the development section and trying to debug the code a bit more. Walking the jboss source with jSwat really will give you a better idea of whats going on. I also sugest some good logging techniques. I'll followup this message with some xml to put in your log4j.xml to log the resources.

- Eric

[ericmolitor]

<!-- logs all resource information at a trace level -->
<!-- including stack trace to identify statement.close issues -->











<appender-ref ref="RESOURCE"/>