Maximum Database connection issue in JBoss 5.1

[amathewjboss1]

Hi,

    We have a big application which uses JBoss 5.1 and is running on Linux servers. Our production environments has 20 web servers (which runs on Apache/Tomcat) which talks to 4 app servers (which runs on JBoss 5.1 in one cluster) with a Oracle 9i database.

   As far as the JBoss configuration is concerned, we are using pooledha invoker as our EJB calls are remote calls. Also <max-pool-size> in the oracle-ds.xml is set to 275. The ha-legacy-jboss-beans.xml has the following setup for the pooledInvokerHA:

  <bean name="PooledInvokerHA"

         class="org.jboss.invocation.pooled.server.PVOPooledInvokerHA">

      <depends>jboss:service=TransactionManager</depends>

      <depends>HAPartition</depends>

      <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss:service=invoker,type=pooledha", exposedInterface=org.jboss.invocation.pooled.server.PooledInvokerMBean.class, registerDirectly=true)</annotation>

      <property name="numAcceptThreads">1</property>

      <property name="maxPoolSize">800</property>

      <property name="clientMaxPoolSize">300</property>

      <property name="socketTimeout">150000</property>

            <property name="serverBindAddress">

         <value-factory bean="ServiceBindingManager" method="getStringBinding"

            parameter="jboss:service=invoker,type=pooledha"/>

      </property>

      <property name="serverBindPort">

         <value-factory bean="ServiceBindingManager" method="getIntBinding"

            parameter="jboss:service=invoker,type=pooledha"/>

      </property>

      <property name="clientConnectAddress">${jboss.bind.address}</property>

      <property name="clientConnectPort">0</property>

      <property name="enableTcpNoDelay">false</property>

      <property name="transactionManagerService">jboss:service=TransactionManager</property>

   </bean>

   The issue we are facing is that when we have an average/high load, we are reaching the "InUseConnectionCount" to 275 very quickly. When we look at the database side, we really don't have any traffic. So once we reached the maximum connection mark, the app servers started to get slow and then FULL GC and then it will crash. An interesting point is that during this situation just recycling the app servers doesn't kill the connections though.i.e we end up in recycling the complete web servers so that all the connections will be released. We spend many hours trying to figure out the issue, but no luck. So any thoughts on this issue will be appreciated.

Thanks

Anil Mathew

[wdfink]

I suppose you have more than 4*275 clients.

After yoiu reach InUseConnectionCount=275 on one instance the next calls to a connection will wait the 'BlockingTimeout' of the DS configuration.

This will often end up in such situations, many threads are waiting for a connection and block memory and resources, the GC can not free memory, the server gets slower and you will have a crash because of too much GC load.

You might

- increase the number of connections

  might not help because you must handle the number of threads in one instance (maybe increase memory)

- add JBoss instances

- limit the number of web sessions

[amathewjboss1]

Thank You so much for the response.

The main problem is what you mentioned above. i.e when this issue happens we DO NOT have 4*275 concurrent request (clients). This is why i really can't figure out how we are reaching this maximum "in use connection count".

We don't have a <blocking-timeout-millis> entry in the ds file, which means it is using the default (i.e 30 seconds).

We did 2 of the above which you mentioned as a workaround. i.e added more jboss instances, limited the number of web sessions. These changes did in fact improved our situation but we still reach the "maximum in use connection count".

Also i wanted to share this link with you - - https://issues.jboss.org/browse/JBAS-4047. I got this as a reply for my posting.

Like it is mentioned in the above Jboss URL:

“The PooledInvokerProxy lock is held while getting an available connection from the pool and while the connection is tested. If the connection is to a local (machine) process, it may not take long for the server ACK to arrive. If the connection is to a remote (machine) process, it could take longer depending on network latency (especially if a wan is between tiers). “

My thought is that when the ACK server call (since we are using PooledInvoker) isslow, many threads will wait for a connection.

Again thanks for your time.

Anil Mathew

[wdfink]

I don't think that the Tomcat->JBoss connection will be the problem.

It might be a problem whether the DB connection is not 'closed' that mean that it is not correct given back to the pool.

Such problems happen if the ResultSet/Statement/Connection is not closed in any circumstances. i.e. if you use the connection to execute your own SQL.

You will find settings and informations in this wiki, also the <track-statements> element in the ds configuration (see this wiki) might help