DS - No managed connections available within blocking timeou

[sc0tt]

Hi,
I have been performance tuning jboss, and have came across an issue related to DataSources. Each time I run a 500 user test, the application "pauses" for 200 seconds or so(no responses are seen). I have tracked this down to two separate issues, the first was with the number of Open file descriptors. I am using RHEL3, and this is set to 1024, for an indiviual process. I have moved this to 2048, on the grounds that each user connection(we are connecting directly to jboss), consumes one connection, and logically since everything in linux is a file, needs one file descriptor. I no longer get the "Too many open files"(note I have checked for increasing numbers of open files, but do not see this increasing over time), effectively I have an application that requires a lot of open file descriptors. Now this is stabalised.

However, I still get the problem with "No managed connections available". I have changed all the relevant DataSource maximum connections from a default of 20, to a new default maximum of 200(changed in the ds.xml for each datasource). I have monitored the application and am getting around 100 data source connections maximum, per data source(ie I should have another 100 "spare").

However, I still have the error that there are no managed connections available.

Can anyone shed any light on what options I might try next to track this down?

I have included a stack trace from the log file.....




Caused by: org.jboss.util.NestedSQLException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] ); - neste
d throwable: (javax.resource.ResourceException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] ))
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:94)
at org.jboss.ejb.txtimer.GeneralPurposeDatabasePersistencePlugin.insertTimer(GeneralPurposeDatabasePersistencePlugin.java:172)
at org.jboss.ejb.txtimer.DatabasePersistencePolicy.insertTimer(DatabasePersistencePolicy.java:122)
... 100 more
Caused by: javax.resource.ResourceException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] )
at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:266)
at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:454)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:324)
at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:301)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:379)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.
java:812)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:88)
... 102 more

[bytor99999]

Not that I am any expert, but what about if the issue is on the database side, like the Database can only create 100 connections/server processes. Ask your DBA as to what that is set to.

Mark

[bytor99999]

But, the other question is why you need to set it that high of 200, How many users are you expecting. If you need 200, then a test with only 500 users is not going to help you test performance on your server, to simulate what your normal traffic would be.

[sc0tt]

Hi,
The number of users(clients) is set at 500. In the XXXXDS.xml(four of them), I have set the max connections to 200. Normally it sits around 100(even with the 500 user load). However, after say, 1 hour, the server itself pauses. No clients are serviced, and it is impossible to run any commands on the server, as it will not respond. This pause can last 200 seconds, after which time the server happily recovers.
I will increase the max DS connections to 750(ie larger than the current user load), just to test. However, I would expect Datasources to be reused, and having a 500 user load with only 100 datasource connections seems quite reasonable.
In terms of the db backend, this is oracle. The oracle db seems quite happy, and simply pauses whilst the application pauses. I have looked into several "statspack" reports for the oracle db during this time, but no "unusual" activity is reported, and nothing seems to be wrong. In summary the oracle server seems perfectly fine.
A stacktrace from the jboss server, indicates that there are several threads waiting on a db connection, yet there seems to be plenty of available headroom, for the datasources to grow, but nothing is growing them?

thanks
Sc0tt....

[4l3xb]

possibilities:
- connection pool leak, http://wiki.jboss.org/wiki/Wiki.jsp?page=CanJBossTellMeWhenIDontCloseAConnection, fix your app
- another resource pool in the system is constraining the number of concurrent connections/threads.
- JBoss not providing the required number of connections in the pool, check bug database, if none match, raise a defect will jboss,
- db server not configured to allow more than 100 connections, speak to dba as suggested,

you've already taken thread dumps and found not all threads are waiting for new db cons. points towards a leak or some other thread pool constraint

get some stats from http://wiki.jboss.org/wiki/Wiki.jsp?page=CanJBossTellMeWhenIDontCloseAConnection

the pause/hang is probably due to the server running out of threads to service the requests (server.xml, maxThreads, default 200 - that needs to be at least 500 if you expect 500 concurrent requests), once this happens there is an overflow number (aceptCount, default 10) of connections that jboss will allow your stress tool to connect to, which will just sit and wait to be serviced also.

as the connection wait timeout is 30 seconds, once the connection pool is exhausted requests will wait 30 seconds until they fail. When you have >100 concurrent clients it may take a long time for all those requests to timeout before the server will respond again (the 200 seconds sounds reasonable). I'd suggest reducing the timeout to something like 200 milliseconds if your db server is on a local gigabit LAN, any longer and it's likely there is a problem in which case you'd want it to fail sooner rather than later in tests. On a LAN you should be able to connect to your db server in a couple of milliseconds.

monitor the number of concurrent connection to Jboss and db server on a 30 second interval and note if/when the connections shoot up. take dumps immediately after, don't wait for the hang.

does the same issue occur for 400,300,200,50 users after 1,2,3 hours? If it's a leak it doesn't matter how many users necessarily, as it may just be a matter of time before the connections run out.

I'd hope some of the above would get you closer to a diagnosis.

[sc0tt]

Hi,
Thanks for the suggestions, have modified the setup to monitor for connection leaks as suggested.
In the meantime...

server.xml.....max connections = 1200 (normally, the number of jboss threads sits around 600 - around 75 from a fresh jboss start, and then a further 500 or so once I load the application with 500 users)


I will drop an update as soon as I have implemented any/all of your suggestions....thanks for the comments...
Sc0tt....

[amjad277]

At the end of each user test you must close the connection, session and producer.
Then you will not get this exceptions.
I had the same problem with JBoss!

Best regards
Amjad

[amjad277]

I had the same problem with JBoss! and I solved it:
At the end of each user test you must close the connection, session and producer.
Then you will not get this exceptions.

Best regards
Amjad