4 Replies Latest reply on May 30, 2012 4:05 AM by galder.zamarreno

    Eviction process don't release connections from pool in 5.1.3CR1?

    jbaris

      Hi everybody.

      We have a web application with ISPN in embedded mode. With ISPN 5.0.1.FINAL all works ok. But recently, have migrated ISPN to 5.1.3CR1 and we are experiencing problems. Basically, the aplicaction throws exceptions like:

       

      2012-03-29 16:12:28,553 ERROR [org.infinispan.loaders.AbstractCacheStore] ISPN000045: Problems encountered while purging expired

      org.infinispan.loaders.CacheLoaderException: This might be related to https://jira.jboss.org/browse/ISPN-604

          at org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:93)

          at org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore.purgeInternal(JdbcStringBasedCacheStore.java:318)

          at org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStore.purgeInternal(JdbcMixedCacheStore.java:130)

          at org.infinispan.loaders.AbstractCacheStore$2.run(AbstractCacheStore.java:106)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:898)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:920)

          at java.lang.Thread.run(Thread.java:736)

      Caused by:

      org.jboss.util.NestedSQLException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] ); - nested throwable: (javax.resource.ResourceException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] ))

          at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:96)

          at org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:90)

          ... 6 more

      Caused by:

      javax.resource.ResourceException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] )

          at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:301)

          at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:500)

          at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:348)

          at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:330)

          at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:423)

          at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:850)

          at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:90)

          at org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:94)

          ... 6 more

       

      Apparently, the eviction process in not releasing the connections from the connections pool (managed by the application server) .

      The solution for avoid this exceptions is to disable the eviction, by adding the following tag to the ispn configuration file:

       

      <infinispan ...>

      ...

      <default>

      ....

      <expiration wakeUpInterval="-1"/>

      </default>

      </infinispan>

       

      Are we doing something wrong or is this is a bug?

      There is the configuration we are using:

       

      db2-jcc-xa-ds.xml

      <?xml version="1.0" encoding="UTF-8"?>

      <datasources>

         <xa-datasource>

              <jndi-name>DB2XADS</jndi-name>

              <xa-datasource-class>com.ibm.db2.jcc.DB2XADataSource</xa-datasource-class>

              <xa-datasource-property name="ServerName">192.168.0.1</xa-datasource-property>

              <xa-datasource-property name="DatabaseName">DB</xa-datasource-property>

              <xa-datasource-property name="User">user</xa-datasource-property>

              <xa-datasource-property name="Password">pass</xa-datasource-property>

              <xa-datasource-property name="PortNumber">50000</xa-datasource-property>

              <xa-datasource-property name="DriverType">4</xa-datasource-property>

              <xa-datasource-property name="CurrentSchema">INFINISPAN</xa-datasource-property>

              <isSameRM-override-value>false</isSameRM-override-value>

              <validation>

                <valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.db2.DB2ValidConnectionChecker"></valid-connection-checker>

                <stale-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.db2.DB2StaleConnectionChecker"></stale-connection-checker>

                <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.db2.DB2ExceptionSorter"></exception-sorter>

              </validation>

              <metadata>

                  <type-mapping>DB2</type-mapping>

              </metadata>

          </xa-datasource>

      </datasources>

       

      infinispan.xml:

      <?xml version="1.0" encoding="UTF-8"?>

      <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

          xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

          xmlns="urn:infinispan:config:5.1">

          <global>

              <globalJmxStatistics enabled="true"

                  cacheManagerName="InfinispanTest" />

              <transport clusterName="InfinispanTest"/>

          </global>

          <default>

              <jmxStatistics enabled="true" />

              <clustering mode="distribution">

                  <hash numOwners="3"/>

                  <sync />          

              </clustering>

              <locking useLockStriping="false" />

              <deadlockDetection enabled="true" spinDuration="500" />

              <transaction

                  transactionManagerLookupClass="org.jboss.cache.JBossTransactionManagerLookup"

                  syncCommitPhase="true" syncRollbackPhase="true" useEagerLocking="false"

                  useSynchronization="false" eagerLockSingleNode="false">

                  <recovery enabled="true" />

              </transaction>

              <loaders passivation="false" shared="true" preload="true">

                  <loader class="org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStore"

                      fetchPersistentState="false" ignoreModifications="false"

                      purgeOnStartup="false">

                      <properties>

                          <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory" />

                          <property name="datasourceJndiLocation" value="java:DB2XADS" />

                          <property name="databaseType" value="DB2"/>

                          <property name="tableNamePrefixForStrings" value="ISPN_MIXED_STR_TABLE" />

                          <property name="tableNamePrefixForBinary" value="ISPN_MIXED_BINARY_TABLE" />

                          <property name="idColumnNameForStrings" value="ID_COLUMN" />

                          <property name="idColumnNameForBinary" value="ID_COLUMN" />

                          <property name="dataColumnNameForStrings" value="DATA_COLUMN" />

                          <property name="dataColumnNameForBinary" value="DATA_COLUMN" />

                          <property name="timestampColumnNameForStrings" value="TIMESTAMP_COLUMN" />

                          <property name="timestampColumnNameForBinary" value="TIMESTAMP_COLUMN" />

                          <property name="timestampColumnTypeForStrings" value="BIGINT" />

                          <property name="timestampColumnTypeForBinary" value="BIGINT" />

                          <property name="idColumnTypeForStrings" value="VARCHAR(255)" />

                          <property name="idColumnTypeForBinary" value="VARCHAR(255)" />

                          <property name="dataColumnTypeForStrings" value="BLOB" />

                          <property name="dataColumnTypeForBinary" value="BLOB" />

                          <property name="dropTableOnExitForStrings" value="false" />

                          <property name="dropTableOnExitForBinary" value="false" />

                          <property name="createTableOnStartForStrings" value="true" />

                          <property name="createTableOnStartForBinary" value="true" />

                          <property name="createTableOnStartForStrings" value="true" />

                          <property name="createTableOnStartForBinary" value="true" />

                      </properties>

                  </loader>

              </loaders>

          </default>

       

          <namedCache name="MYCACHE"/>

       

      </infinispan>

      Thanks

        • 2. Re: Eviction process don't release connections from pool in 5.1.3CR1?
          jbaris

          Galder, thanks for replying

          I made these changes, but 4 tests fail (errors are bellow).

          I tried some alternatives with no success.

           

          Have you any other suggestions?

           

          Thanks

           

          -------------------------------------------------------------------------------------------------------------

          loaders.jdbc.mixed.JdbcMixedCacheStoreTest

          Tests passed/Failed/Skipped:    5/2/0

          Started on:    Mon May 28 14:22:45 ART 2012

          Total time:    9 seconds (9068 ms)

          Included groups:   

          Excluded groups:   

           

          (Hover the method name to see the test class name)

           

          FAILED TESTS

          Test method     Exception     Time (seconds)     Instance

          testPurgeExpired

          Test class: org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest    

           

          java.lang.AssertionError: Expected 0 rows, actual value is 1

              at org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest.assertBinaryRowCount(JdbcMixedCacheStoreTest.java:221)

              at org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest.assertRowCounts(JdbcMixedCacheStoreTest.java:204)

              at org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest.testPurgeExpired(JdbcMixedCacheStoreTest.java:185)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:662)

          ... Removed 17 stack frames

           

          Click to show all stack frames     1     org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest@6286e3

          testPurgeExpiredWithRemainingEntries

          Test class: org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest    

           

          java.lang.AssertionError: Expected 1 rows, actual value is 2

              at org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest.assertBinaryRowCount(JdbcMixedCacheStoreTest.java:221)

              at org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest.assertRowCounts(JdbcMixedCacheStoreTest.java:204)

              at org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest.testPurgeExpiredWithRemainingEntries(JdbcMixedCacheStoreTest.java:200)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:662)

          ... Removed 17 stack frames

           

          1     org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest@6286e3

          -------------------------------------------------------------------------------------------------------------

          loaders.jdbc.mixed.JdbcMixedCacheStoreVamTest

          Tests passed/Failed/Skipped:    5/2/0

          Started on:    Mon May 28 14:22:45 ART 2012

          Total time:    9 seconds (9445 ms)

          Included groups:   

          Excluded groups:   

           

          (Hover the method name to see the test class name)

           

          FAILED TESTS

          Test method     Exception     Time (seconds)     Instance

          testPurgeExpired

          Test class: org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreVamTest    

           

          java.lang.AssertionError: Expected 0 rows, actual value is 1

              at org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest.assertBinaryRowCount(JdbcMixedCacheStoreTest.java:221)

              at org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest.assertRowCounts(JdbcMixedCacheStoreTest.java:204)

              at org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest.testPurgeExpired(JdbcMixedCacheStoreTest.java:185)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:662)

          ... Removed 17 stack frames

           

          Click to show all stack frames     1     org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreVamTest@31028a

          testPurgeExpiredWithRemainingEntries

          Test class: org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreVamTest    

           

          java.lang.AssertionError: Expected 1 rows, actual value is 2

              at org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest.assertBinaryRowCount(JdbcMixedCacheStoreTest.java:221)

              at org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest.assertRowCounts(JdbcMixedCacheStoreTest.java:204)

              at org.infinispan.loaders.jdbc.mixed.JdbcMixedCacheStoreTest.testPurgeExpiredWithRemainingEntries(JdbcMixedCacheStoreTest.java:200)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:662)

          ... Removed 17 stack frames


          • 3. Re: Eviction process don't release connections from pool in 5.1.3CR1?
            galder.zamarreno

            You're right, my suggestion won't work as expected. I've tried to look for some leaks and might have found couple of them.

             

            Can you try building the following branch and use the resulting jars? you'd need core and cachestore/jdbc jars at the bare minimum.

             

            If the issue still present, can you provide a test case to replicate your original issue?

            • 4. Re: Eviction process don't release connections from pool in 5.1.3CR1?
              galder.zamarreno