1 2 Previous Next 19 Replies Latest reply on May 21, 2014 7:34 AM by vinayak_pokharkar

    Connection pool is shutdown periodically

    didier2011

      Hi,

       

      I've got some serious problems with the connection pooling, we've got a five hour test suite for user acceptance testing, every couple of minutes, it's not possible to get a managed connection. The pool is shutdown. Some minutes after the exception the pool seems to recover, till the next exception (sometimes after five minutes, sometimes after an hour)!

       

      What's the problem? Is this a known problem on 7.1 Beta? What could cause this?

       

       

      11:49:54,528 DEBUG [org.hibernate.hql.internal.ast.QueryTranslatorImpl] (ajp--127.0.0.1-21437-2) org.hibernate.internal.CoreMessageLogger_$logger

      11:49:54,528 DEBUG [org.hibernate.hql.internal.ast.ErrorCounter] (ajp--127.0.0.1-21437-2) org.hibernate.internal.CoreMessageLogger_$logger

      11:49:54,529 DEBUG [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (ajp--127.0.0.1-21437-2) org.hibernate.internal.CoreMessageLogger_$logger

      11:49:54,529 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (JBossConnectionValidator) run: ConnectionValidator notifying pools, interval: 2

      11:49:54,529 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (JBossConnectionValidator) my.ds.core-pool: emptyManagedConnectionPool(org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool@bcbf5c)

      11:49:54,529 DEBUG [org.jboss.jca.core.connectionmanager.pool.validator.ConnectionValidator] (JBossConnectionValidator) internalUnregisterPool: setting interval to Long.MAX_VALUE

      11:49:54,530 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ajp--127.0.0.1-21437-2) my.ds.core-pool: getConnection(null, null) [1/75]

      11:49:54,531 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (JBossConnectionValidator) run: ConnectionValidator notifying pools, interval: 2

      11:49:54,533 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (JBossConnectionValidator) run: ConnectionValidator notifying pools, interval: 2

      11:49:54,535 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (JBossConnectionValidator) run: ConnectionValidator notifying pools, interval: 2

      11:49:54,537 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (JBossConnectionValidator) run: ConnectionValidator notifying pools, interval: 2

      11:49:54,539 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (JBossConnectionValidator) run: ConnectionValidator notifying pools, interval: 2

      11:49:54,541 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (JBossConnectionValidator) run: ConnectionValidator notifying pools, interval: 2

      11:49:54,531 DEBUG [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (ajp--127.0.0.1-21437-2) Could not open connection [n/a]: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/my.ds.core

              at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:137)

              at org.hibernate.service.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:141) [hibernate-core-4.0.0.CR6.ja

      r:4.0.0.CR6]

              at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:276) [hibernate-core-4.0.0.CR6.jar:4.0.0.CR6]

              at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:299) [hibernate-core-4.0.0.CR6.jar:4.0.0.CR6]

              at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169) [hibernate-core-4.0.0.CR6.jar:4.0.0.CR6]

              at org.hibernate.engine.jdbc.internal.proxy.ConnectionProxyHandler.extractPhysicalConnection(ConnectionProxyHandler.java:82) [hibernate-core-4.0.0.CR6.jar:4.0.0.CR6]

              at org.hibernate.engine.jdbc.internal.proxy.ConnectionProxyHandler.continueInvocation(ConnectionProxyHandler.java:138) [hibernate-core-4.0.0.CR6.jar:4.0.0.CR6]

              at org.hibernate.engine.jdbc.internal.proxy.AbstractProxyHandler.invoke(AbstractProxyHandler.java:81) [hibernate-core-4.0.0.CR6.jar:4.0.0.CR6]

              at $Proxy81.prepareStatement(Unknown Source)    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:149) [hibernate-core-4

      .0.0.CR6.jar:4.0.0.CR6]

              at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:178) [hibernate-core-4.0.0.CR6.jar:4

      .0.0.CR6]

              at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:147) [hibernate-core-4.0.0.CR6.jar:4.0.0.CR6]

              at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1739) [hibernate-core-4.0.0.CR6.jar:4.0.0.CR6]

              at org.hibernate.loader.Loader.doQuery(Loader.java:828) [hibernate-core-4.0.0.CR6.jar:4.0.0.CR6]

              at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:289) [hibernate-core-4.0.0.CR6.jar:4.0.0.CR6]

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]

      ...

             at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:151) [jboss-as-web-7.1.0.Alpha2-SNAPSHOT.jar:7.1.0.Alpha2-SNAPSHOT]

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]

              at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]

              at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]

              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.3.Final.jar:7.1.0.Alpha2-SNAPSHOT]

              at java.lang.Thread.run(Thread.java:662) [:1.6.0_26]

      Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/my.ds.core

              at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:375)

              at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:367)

              at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:448)

              at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:129)

              ... 111 more

      Caused by: javax.resource.spi.RetryableUnavailableException: The pool has been shutdown

              at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:264)

              at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getTransactionNewConnection(AbstractPool.java:492)

              at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:373)

              at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:332)

              ... 114 more

        • 1. Re: Connection pool is shutdown periodically
          jesper.pedersen

          Can you post your configuration ? You are likely hitting JBJCA-701, but the pool should never be stopped without an explicit command to do so... Does allocation-retry help ?

          1 of 1 people found this helpful
          • 2. Re: Connection pool is shutdown periodically
            didier2011

            We just use a plain vanilla configuration:

             

            for jca:

             

            <subsystem xmlns="urn:jboss:domain:jca:1.0">

                <archive-validation enabled="false"/>

                <bean-validation enabled="false"/>

                <default-workmanager>

                    <short-running-threads blocking="true" allow-core-timeout="false">

                        <core-threads count="10" per-cpu="20"/>

                        <queue-length count="10" per-cpu="20"/>

                        <max-threads count="10" per-cpu="20"/>

                        <keepalive-time time="10" unit="seconds"/>

                    </short-running-threads>

                    <long-running-threads blocking="true" allow-core-timeout="false">

                        <core-threads count="10" per-cpu="20"/>

                        <queue-length count="10" per-cpu="20"/>

                        <max-threads count="10" per-cpu="20"/>

                        <keepalive-time time="10" unit="seconds"/>

                    </long-running-threads>

                </default-workmanager>

            </subsystem>

             

            and for datasources

             

            <subsystem xmlns="urn:jboss:domain:datasources:1.0">

                <datasources>

                    <xa-datasource jndi-name="java:/my.ds.core" pool-name="my.ds.core-pool" enabled="true" use-java-context="true">

                        <xa-datasource-property name="DatabaseName">

                            myDatabaseName

                        </xa-datasource-property>

                        <xa-datasource-property name="User">

                            myUserName

                        </xa-datasource-property>

                        <xa-datasource-property name="ServerName">

                            myServerName

                        </xa-datasource-property>

                        <xa-datasource-property name="PortNumber">

                            myPortNumber

                        </xa-datasource-property>

                        <xa-datasource-property name="Password">

                            myPassword

                        </xa-datasource-property>

                        <xa-datasource-class>

                            org.postgresql.xa.PGXADataSource

                        </xa-datasource-class>

                        <driver>

                            postgresql-jdbc4

                        </driver>

                        <transaction-isolation>

                            TRANSACTION_SERIALIZABLE

                        </transaction-isolation>

                        <xa-pool>

                            <min-pool-size>

                                5

                            </min-pool-size>

                            <max-pool-size>

                                75

                            </max-pool-size>

                        </xa-pool>

                        <security>

                            <user-name>

                                myUserName

                            </user-name>

                            <password>

                                myPassword

                            </password>

                        </security>

                        <validation>

                            <check-valid-connection-sql>

                                select 1

                            </check-valid-connection-sql>

                        </validation>

                        <timeout>

                            <idle-timeout-minutes>

                                5

                            </idle-timeout-minutes>

                        </timeout>

                        <statement>

                            <track-statements>

                                TRUE

                            </track-statements>

                        </statement>

                    </xa-datasource>                                           

                    <drivers>

                        <driver name="postgresql-jdbc4" module="org.postgresql"/>

                    </drivers>

                </datasources>

            </subsystem>

             

            and for login-module:

            <security-domain name="my-security-domain" cache-type="default">

                <authentication>

                    <login-module code="MyDatabaseServerLoginModule" flag="required">

                        <module-option name="debug" value="true"/>

                        <module-option name="dsJndiName" value="java:/my.ds.core"/>

                        <module-option name="principalsQuery" value="MY PrinicipalStatement"/>

                        <module-option name="rolesQuery" value="myRolesQuery"/>

                        <module-option name="hashAlgorithm" value="MD5"/>

                        <module-option name="hashEncoding" value="base64"/>

                        <module-option name="hashCharset" value="UTF-8"/>

                        <module-option name="unauthenticatedIdentity" value="nobody"/>

                    </login-module>

                </authentication>

            </security-domain>

             

            I haven't used "allocation-retry" till now, I'll give it a try...

            • 3. Re: Connection pool is shutdown periodically
              smarlow

              Is this in a test environment or production?  Its possible that capturing additional statistics, could be helpful.  This might be gathering the statistics for the managed connection pool once a minute and correlating the above error with the trend statistics prior/during/after. 

              • 4. Re: Connection pool is shutdown periodically
                smarlow

                I would try increasing idle-timeout-minutes to 60 and see if the problem moves to once an hour, during usability testing.  The workaround for JBJCA-701, is to multiply idle-timeout-minutes by 1000, so better to try idle-timeout-minutes=60000.

                • 5. Re: Connection pool is shutdown periodically
                  didier2011

                  Thank you, "idle-timeout-minutes=60000" solved the problem for my test suite,

                  But I hope, somebody checks, why the pool is shutdown, if idle-timeout-minutes=5

                  • 6. Re: Connection pool is shutdown periodically
                    jesper.pedersen

                    Can you try and attach a TRACE log of the failure (org.jboss.jca) ?

                    • 7. Re: Connection pool is shutdown periodically
                      jesper.pedersen

                      I have an idea - can you see if <use-strict-min>true</..> and <min-pool-size>1</..> works ?

                      1 of 1 people found this helpful
                      • 8. Re: Connection pool is shutdown periodically
                        didier2011

                        Ok, I will run the suite this night with your proposed settings, tomorrow more

                        • 9. Re: Connection pool is shutdown periodically
                          jesper.pedersen

                          The problem should also go away if you upgrade to 1.0.7-SNAPSHOT using the upgrader from IronJacamar 1.1 series and http://docs.jboss.org/ironjacamar/userguide/1.1/en-US/html/installation.html#installas

                          • 10. Re: Connection pool is shutdown periodically
                            didier2011

                            The suite runs successfully with

                            <use-strict-min>true</..> and <min-pool-size>1</..>

                            • 11. Re: Connection pool is shutdown periodically
                              vinayak_pokharkar

                              Hi,

                               

                                   I have application running on Jboss 5.1 appserver which is using xa datastore oracle 10g.

                                   I am also getting below exception periodically for almost 1 minute and then it get recover automatically.

                               

                              Caused by: org.jboss.resource.connectionmanager.RetryableResourceException: The pool has been shutdown

                                at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:202)

                                at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:679)

                                at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)  ... 59 more

                               

                                  Can you please help me to resolve the above exception ?

                               

                               

                              Thanks,
                              Vinayak

                              • 12. Re: Connection pool is shutdown periodically
                                kuldeep11

                                As, suggested by Dieter, You can try same parameter in Jboss 5.1 as well.

                                 

                                <use-strict-min>true</..> and <min-pool-size>1</..>

                                • 13. Re: Connection pool is shutdown periodically
                                  vinayak_pokharkar

                                  Thanks Kuldeep for reply.

                                  Currently I am having below configuration in my *-ds.xml.

                                   

                                   

                                      <xa-datasource>

                                          <jndi-name>DataSource</jndi-name>

                                          <track-connection-by-tx/>

                                          <xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</xa-datasource-class>

                                          <isSameRM-override-value>false</isSameRM-override-value>

                                          <xa-datasource-property name="URL">url</xa-datasource-property>

                                          <security-domain>EncryptDBPassword</security-domain>

                                          <valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker</valid-connection-checker-class-name>

                                          <exception-sorter-class-name>

                                              org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter

                                          </exception-sorter-class-name>

                                          <min-pool-size>100</min-pool-size>

                                          <max-pool-size>300</max-pool-size>

                                    <query-timeout>900</query-timeout>

                                          <no-tx-separate-pools/>

                                      </xa-datasource>

                                   

                                  So shallI go for those changes ? Since my "min-pool-size" is set 100.

                                   

                                  Thanks,

                                  Vinayak

                                  • 14. Re: Connection pool is shutdown periodically
                                    kuldeep11

                                    Setting use-strict-min to true will make sure that your pool will never drop below your min-pool-size. You can keep your min-pool-size as it is and make use-strict-min to true.

                                    then see if it works.

                                     

                                    Thanks!

                                    1 2 Previous Next