3 Replies Latest reply on Nov 25, 2008 7:40 AM by manik

    java.sql.SQLException: ORA-00001: unique constraint (CCC_12.

    system.out

      We are experiencing exception in database layer (in production), upon increasing the load on JBoss Cache. The system is working fine in normal load (one message every half a second coming from 5 different thread). Here is the exception:

      The Isolation level is REPEATABLE_READ, however there is only thread looking at each node, i.e. each message with unique ID will be dealt with one thread only, and the ID is included in the FQN.


      I also came across this related post, but that didn't help :(
      http://www.jboss.com/index.html?module=bb&op=viewtopic&t=96450

      Any idea? Thanks in advance!
      -mike

      2008-11-17 07:43:00,907 ERROR (Timer-4) [org.jboss.cache.loader.JDBCCacheLoader:119] Failed to insert node: ORA-00001: unique constraint (12.JBOSSCACHE_PK) violated

      java.sql.SQLException: ORA-00001: unique constraint (CCC_12.JBOSSCACHE_PK) violated
      at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
      at oracle.jdbc.driver.T2CConnection.checkError(T2CConnection.java:676)
      at oracle.jdbc.driver.T2CConnection.checkError(T2CConnection.java:602)
      at oracle.jdbc.driver.T2CPreparedStatement.executeForDescribe(T2CPreparedStatement.java:571)
      at oracle.jdbc.driver.T2CPreparedStatement.executeForRows(T2CPreparedStatement.java:764)
      at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1170)
      at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3339)
      at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3423)
      at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:251)
      at org.jboss.cache.loader.JDBCCacheLoader.insertNode(JDBCCacheLoader.java:969)
      at org.jboss.cache.loader.JDBCCacheLoader.put(JDBCCacheLoader.java:913)
      at org.jboss.cache.loader.JDBCCacheLoader.put(JDBCCacheLoader.java:341)
      at org.jboss.cache.interceptors.PassivationInterceptor.invoke(PassivationInterceptor.java:63)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:197)
      at org.jboss.cache.interceptors.ActivationInterceptor.invoke(ActivationInterceptor.java:59)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.OptimisticReplicationInterceptor.invoke(OptimisticReplicationInterceptor.java:136)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345)
      at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:179)
      at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520)
      at org.jboss.cache.TreeCache.evict(TreeCache.java:3716)
      at org.jboss.cache.aop.PojoCache.evict(PojoCache.java:209)
      at org.jboss.cache.eviction.BaseEvictionPolicy.evict(BaseEvictionPolicy.java:34)
      at org.jboss.cache.eviction.BaseEvictionAlgorithm.evictCacheNode(BaseEvictionAlgorithm.java:210)
      at org.jboss.cache.eviction.LRUAlgorithm.evict(LRUAlgorithm.java:90)
      at org.jboss.cache.eviction.LRUAlgorithm.prune(LRUAlgorithm.java:121)
      at org.jboss.cache.eviction.BaseEvictionAlgorithm.process(BaseEvictionAlgorithm.java:98)
      at org.jboss.cache.eviction.EvictionTimerTask.run(EvictionTimerTask.java:80)
      at java.util.TimerThread.mainLoop(Timer.java:512)
      at java.util.TimerThread.run(Timer.java:462)


      and here is the cache configuration:
      <?xml version="1.0" encoding="UTF-8"?>
      
      <!-- ===================================================================== -->
      <!-- -->
      <!-- Customized TreeCache Service Configuration for Tomcat 5 Clustering -->
      <!-- -->
      <!-- ===================================================================== -->
      
      <server>
      
       <!-- ==================================================================== -->
       <!-- Defines TreeCache configuration -->
       <!-- ==================================================================== -->
      
       <!-- Note we are using TreeCacheAop -->
       <mbean code="org.jboss.cache.aop.TreeCacheAop" name="jboss.cache:service=TomcatClusteringCache">
       <depends>jboss.jca:service=DataSourceBinding,name=OracleDS</depends>
       <depends>jboss:service=Naming</depends>
       <depends>jboss:service=TransactionManager</depends>
       <!-- We need the AspectDeployer to deploy our FIELD granularity aspects -->
       <depends>jboss.aop:service=AspectDeployer</depends>
      
       <!-- Name of cluster. Needs to be the same for all nodes in the
       cluster, in order to find each other
       -->
       <attribute name="ClusterName">Tomcat-${jboss.partition.name:Cluster}</attribute>
      
       <!--
       Isolation level : SERIALIZABLE
       REPEATABLE_READ (default)
       READ_COMMITTED
       READ_UNCOMMITTED
       NONE
       -->
       <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
      
       <!-- add for bug number 10819 -->
       <attribute name="NodeLockingScheme">OPTIMISTIC</attribute>
       <attribute name="CacheMode">REPL_ASYNC</attribute>
       <attribute name="UseRegionBasedMarshalling">false</attribute>
      
       <!-- Whether or not the entire tree is inactive upon startup, only
       responding to replication messages after activateRegion() is
       called to activate one or more parts of the tree when a webapp is
       deployed. Must have the same value as "UseRegionBasedMarshalling".
       -->
       <attribute name="InactiveOnStartup">false</attribute>
      
       <!-- Make sure to specify BatchModeTransactionManager only! -->
       <attribute name="TransactionManagerLookupClass">org.jboss.cache.BatchModeTransactionManagerLookup</attribute>
       <attribute name="ClusterConfig">
       <config>
       <TCP bind_addr="localhost" start_port="7810" loopback="true" tcp_nodelay="false" down_thread="false" up_thread="false" />
       <TCPPING initial_hosts="localhost[7810],localhost[7830]" port_range="3" timeout="3500" num_initial_members="3" up_thread="false" down_thread="false" />
       <MERGE2 min_interval="5000" max_interval="10000" up_thread="false" down_thread="false" />
       <FD_SOCK down_thread="false" up_thread="false" />
       <FD shun="true" up_thread="false" down_thread="false" timeout="10000" max_tries="5" />
       <VERIFY_SUSPECT timeout="1500" down_thread="false" up_thread="false" />
       <pbcast.NAKACK down_thread="false" up_thread="false" gc_lag="100" retransmit_timeout="3000" />
       <pbcast.STABLE desired_avg_gossip="20000" down_thread="false" up_thread="false" />
       <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="true" print_local_addr="true" down_thread="false" up_thread="false" />
       <FC max_credits="2000000" down_thread="false" up_thread="false" min_threshold="0.10" />
       <FRAG2 frag_size="60000" down_thread="false" up_thread="false" />
       <pbcast.STATE_TRANSFER up_thread="false" down_thread="false" />
       </config>
       </attribute>
      
       <attribute name="SyncReplTimeout">20000</attribute>
      
       <!-- Max number of milliseconds to wait for a lock acquisition -->
       <attribute name="LockAcquisitionTimeout">15000</attribute>
      
       <attribute name="BuddyReplicationConfig">
       <config>
       <buddyReplicationEnabled>false</buddyReplicationEnabled>
       <buddyLocatorClass>org.jboss.cache.buddyreplication.NextMemberBuddyLocator</buddyLocatorClass>
       <buddyLocatorProperties>
       numBuddies = 1
       ignoreColocatedBuddies = true
       </buddyLocatorProperties>
      
       <buddyPoolName>default</buddyPoolName>
       <buddyCommunicationTimeout>2000</buddyCommunicationTimeout>
      
       <autoDataGravitation>false</autoDataGravitation>
       <dataGravitationRemoveOnFind>true</dataGravitationRemoveOnFind>
       <dataGravitationSearchBackupTrees>true</dataGravitationSearchBackupTrees>
      
       </config>
       </attribute>
       <attribute name="EvictionPolicyClass">org.jboss.cache.aop.eviction.AopLRUPolicy</attribute>
      
       <!-- Specific eviction policy configurations. This is LRU -->
       <attribute name="EvictionPolicyConfig">
       <config>
       <attribute name="wakeUpIntervalSeconds">5</attribute>
       <!-- Cache wide default -->
       <region name="/_default_">
       <attribute name="maxNodes">5000</attribute>
       <attribute name="timeToLiveSeconds">600</attribute>
       </region>
       </config>
       </attribute>
      
       <!-- New 1.3.x cache loader config block -->
       <attribute name="CacheLoaderConfiguration">
       <config>
       <passivation>false</passivation>
       <shared>true</shared>
       <cacheloader>
       <class>
       org.jboss.cache.loader.JDBCCacheLoader
       </class>
       <properties>
       cache.jdbc.datasource=java:/OracleDS
       cache.jdbc.table.drop=false
       </properties>
       <fetchPersistentState>true</fetchPersistentState>
       <ignoreModifications>false</ignoreModifications>
       <purgeOnStartup>false</purgeOnStartup>
       </cacheloader>
       </config>
       </attribute>
      
       </mbean>
      </server>


        • 1. Re: java.sql.SQLException: ORA-00001: unique constraint (CCC
          system.out

          Isn't this related to eviction policy, as the EvictionTimerTask will be executed in a different thread? I thought Isolation Level REPEATABLE_READ will take care of dead-lock !


          at org.jboss.cache.TreeCache.evict(TreeCache.java:3716)
          at org.jboss.cache.aop.PojoCache.evict(PojoCache.java:209)
          at org.jboss.cache.eviction.BaseEvictionPolicy.evict(BaseEvictionPolicy.java:34)
          at org.jboss.cache.eviction.BaseEvictionAlgorithm.evictCacheNode(BaseEvictionAlgorithm.java:210)
          at org.jboss.cache.eviction.LRUAlgorithm.evict(LRUAlgorithm.java:90)
          at org.jboss.cache.eviction.LRUAlgorithm.prune(LRUAlgorithm.java:121)
          at org.jboss.cache.eviction.BaseEvictionAlgorithm.process(BaseEvictionAlgorithm.java:98)
          at org.jboss.cache.eviction.EvictionTimerTask.run(EvictionTimerTask.java:80)
          at java.util.TimerThread.mainLoop(Timer.java:512)
          at java.util.TimerThread.run(Timer.java:462)


          • 2. Re: java.sql.SQLException: ORA-00001: unique constraint (CCC
            system.out

            The key/value is ID/collection

            Also, about REPEATABLE_READ

            This level prevents "non-repeatable read" but it does not prevent the so-called "phantom read" where new data can be inserted into the tree from the other transaction


            What happens if evictTask is trying to persist the collection Object, while an incoming message has been added to the collection ? I think by
            default the cache loader runs in async mode. i.e.
            <!-- whether the cache loader writes are asynchronous -->
             <async>true</async>

            If that is correct, maybe the solution would be:
            1. set the cache loader to be synchronous
            2. set the isolation level to SERIALIZABLE


            does it make sense?



            • 3. Re: java.sql.SQLException: ORA-00001: unique constraint (CCC
              manik

              Looks like you're seeing http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4173132#4173132. Fixed in JBCACHE-1410