4 Replies Latest reply on Dec 16, 2006 1:35 AM by Archanaa Panda

    Problem with JBoss JDBCCacheLoader

    Archanaa Panda Newbie

      Hello,

      We are using JBoss Cache 1.3 for our project which is under development. Recently while doing some performance testing we found a problem with the JDBC Cache Loader (we are using an Oracle database as the backing store for our JBoss Cache). When we run with multiple threads or sometimes even one thread, we get the following exception:

      java.lang.RuntimeException: java.lang.IllegalStateException: Failed to insert node: ORA-00001: unique constraint (FASTDEV.JBOSSCACHE_PK) violated at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:4810)
      at org.jboss.cache.TreeCache.put(TreeCache.java:3242)
      at com.sisl.ashleycomm.cache.CacheTimeoutManager.putCacheable(CacheTimeoutManager.java:452)
      at com.sisl.ashleycomm.cache.push.command.MobilePushCacheTimeoutManager.putCacheable(MobilePushCacheTimeoutManager.java:168)
      at com.sisl.ashleycomm.cache.DefaultCacheableVersionProxy.processEvent(DefaultCacheableVersionProxy.java:134)
      at com.sisl.ashleycomm.cache.push.command.MobilePushCacheTimeoutManager.getMobilePushCommandsForGprsSend


      Caused by: java.lang.IllegalStateException: Failed to insert node: ORA-00001: unique constraint (FASTDEV.JBOSSCACHE_PK) violated
      at org.jboss.cache.loader.JDBCCacheLoader.insertNode(JDBCCacheLoader.java:982)
      at org.jboss.cache.loader.JDBCCacheLoader.put(JDBCCacheLoader.java:325)
      at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:202)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
      at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:187)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
      at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
      at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:32)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
      at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:328)
      at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:139)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
      at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:153)
      at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:4804)


      Actually the scenario is like this:
      A thread gets a node on the cache and then does some modification on the node and tries to put it back on the cache.
      There may be multiple threads accessing the same node.

      Looking at the source code of JBoss Cache 1.3, I find that when put() is called on JDBCCacheLoader, loadNode() method is called. Apparantly this is returning null so the insertNode() is being called. But then when it tries to do the insertNode, it gets the PK violated error which indicates the row was existing in the database already!!!

      What are the precautions we need to take to avoid this problem? Is our setting wrong?

      A quick reply would be very much appreciated. Please let me know if there are more details required for diagnosing the problem

      Thanks and Regards,
      Archanaa Panda

      Following are the details of our cache config xml file-

      <attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute>
       <attribute name="IsolationLevel">READ_COMMITTED</attribute>
      <attribute name="CacheMode">REPL_SYNC</attribute>
      
      
      <attribute name="CacheLoaderConfiguration">
       <config>
       <passivation>false</passivation>
       <preload>/</preload>
       <shared>true</shared>
       <cacheloader>
       <class>org.jboss.cache.loader.JDBCCacheLoader</class>
       <properties>
       cache.jdbc.table.name=jbosscache
       cache.jdbc.table.create=true
       cache.jdbc.table.drop=false
       cache.jdbc.table.primarykey=jbosscache_pk
       cache.jdbc.fqn.column=fqn
       cache.jdbc.fqn.type=varchar(255)
       cache.jdbc.node.column=node
       cache.jdbc.node.type=blob
       cache.jdbc.parent.column=parent
       cache.jdbc.driver=oracle.jdbc.driver.OracleDriver
       cache.jdbc.url=jdbc:oracle:thin:@132.186.106.26:1541:ACRMDEV
       cache.jdbc.user=FASTDEV
       cache.jdbc.password=fastdev123$
       </properties>
       <async>false</async>
       <fetchPersistentState>true</fetchPersistentState>
       <ignoreModifications>false</ignoreModifications>
       </cacheloader>
       </config>
       </attribute>



        • 1. Re: Problem with JBoss JDBCCacheLoader
          Hany Mesha Novice

          I think you have a race condition going on here stems from your application doing this call com.sisl.ashleycomm.cache.CacheTimeoutManager.putCacheable(CacheTimeoutManager.java:452)

          Are you using eviction policy? Can you post the entire JBossCache configuration you're using?

          what isolation level you're using?

          • 2. Re: Problem with JBoss JDBCCacheLoader
            Archanaa Panda Newbie

            Thank you so very much for looking into the problem. I was on leave as i had fallen ill and i just saw your reply. Our isolation level is REPEATABLE_READ and we are using REPL_SYNC cache mode

            Please find below the complete configuration-

            <?xml version="1.0" encoding="UTF-8"?>
            
            <!-- ===================================================================== -->
            <!-- -->
            <!-- Sample TreeCache Service Configuration -->
            <!-- -->
            <!-- ===================================================================== -->
            
            <server>
            
             <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/>
            
            
             <!-- ==================================================================== -->
             <!-- Defines TreeCache configuration -->
             <!-- ==================================================================== -->
            
             <mbean code="org.jboss.cache.TreeCache"
             name="jboss.cache:service=TreeCache">
            
             <depends>jboss:service=Naming</depends>
             <depends>jboss:service=TransactionManager</depends>
            
             <!--
             Configure the TransactionManager
             -->
             <attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute>
            
             <!--
             Isolation level : SERIALIZABLE
             REPEATABLE_READ (default)
             READ_COMMITTED
             READ_UNCOMMITTED
             NONE
             -->
             <!--
             <attribute name="NodeLockingScheme">OPTIMISTIC</attribute>
             -->
             <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
            
             <!--
             Valid modes are LOCAL
             REPL_ASYNC
             REPL_SYNC
             INVALIDATION_ASYNC
             INVALIDATION_SYNC
             -->
             <attribute name="CacheMode">REPL_SYNC</attribute>
            
             <!--
             Just used for async repl: use a replication queue
             -->
             <attribute name="UseReplQueue">false</attribute>
            
             <!--
             Replication interval for replication queue (in ms)
             -->
             <attribute name="ReplQueueInterval">0</attribute>
            
             <!--
             Max number of elements which trigger replication
             -->
             <attribute name="ReplQueueMaxElements">0</attribute>
            
             <!-- Name of cluster. Needs to be the same for all clusters, in order
             to find each other
             -->
             <attribute name="ClusterName">TreeCache-Cluster</attribute>
            
             <!-- JGroups protocol stack properties. Can also be a URL,
             e.g. file:/home/bela/default.xml
             <attribute name="ClusterProperties"></attribute>
             -->
            
             <attribute name="ClusterConfig">
             <config>
             <!-- UDP: if you have a multihomed machine,
             set the bind_addr attribute to the appropriate NIC IP address, e.g bind_addr="192.168.0.2"
             -->
             <!-- UDP: On Windows machines, because of the media sense feature
             being broken with multicast (even after disabling media sense)
             set the loopback attribute to true -->
             <UDP mcast_addr="228.1.2.3" mcast_port="48866"
             ip_ttl="64" ip_mcast="true"
             mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
             ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
             loopback="false"/>
             <PING timeout="2000" num_initial_members="3"
             up_thread="false" down_thread="false"/>
             <MERGE2 min_interval="10000" max_interval="20000"/>
             <!-- <FD shun="true" up_thread="true" down_thread="true" />-->
             <FD_SOCK/>
             <VERIFY_SUSPECT timeout="1500"
             up_thread="false" down_thread="false"/>
             <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
             max_xmit_size="8192" up_thread="false" down_thread="false"/>
             <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
             down_thread="false"/>
             <pbcast.STABLE desired_avg_gossip="20000"
             up_thread="false" down_thread="false"/>
             <FRAG frag_size="8192"
             down_thread="false" up_thread="false"/>
             <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
             shun="true" print_local_addr="true"/>
             <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
             </config>
             </attribute>
            
            
             <!--
             Whether or not to fetch state on joining a cluster
             NOTE this used to be called FetchStateOnStartup and has been renamed to be more descriptive.
             -->
             <attribute name="FetchInMemoryState">true</attribute>
            
             <!--
             The max amount of time (in milliseconds) we wait until the
             initial state (ie. the contents of the cache) are retrieved from
             existing members in a clustered environment
             -->
             <attribute name="InitialStateRetrievalTimeout">15000</attribute>
            
             <!--
             Number of milliseconds to wait until all responses for a
             synchronous call have been received.
             -->
             <attribute name="SyncReplTimeout">15000</attribute>
            
             <!-- Max number of milliseconds to wait for a lock acquisition -->
             <attribute name="LockAcquisitionTimeout">10000</attribute>
            
             <!-- Name of the eviction policy class. -->
             <attribute name="EvictionPolicyClass"></attribute>
            
             <!--
             Indicate whether to use marshalling or not. Set this to true if you are running under a scoped
             class loader, e.g., inside an application server. Default is "false".
             -->
             <attribute name="UseMarshalling">false</attribute>
             </mbean>
            
            
             <!-- Uncomment to get a graphical view of the TreeCache MBean above -->
             <!-- <mbean code="org.jboss.cache.TreeCacheView" name="jboss.cache:service=TreeCacheView">-->
             <!-- <depends>jboss.cache:service=TreeCache</depends>-->
             <!-- <attribute name="CacheService">jboss.cache:service=TreeCache</attribute>-->
             <!-- </mbean>-->
            
            
            </server>


            • 3. Re: Problem with JBoss JDBCCacheLoader
              Hany Mesha Novice

              Are you sure this's your current cache configuration? It doesn't have even the JDBC cache loader configuration which was in your first post?!!

              Anyway, I think based on the your first post cache configuration that more than one thread is accessing the data with read committed isolation level which means that Non-repeatable reads may occur. That would result in data retrieved in a SELECT statement may be modified by some other transaction when it commits.


              Actually the scenario is like this:
              A thread gets a node on the cache and then does some modification on the node and tries to put it back on the cache.
              There may be multiple threads accessing the same node.

              So when you do a get on the node (SELECT) the entry wasn't there and therefore loadNode() returns null but then when the transcation commit to the database trying to put the node (INSERT) rather than (UPDATE) the entry was already added by another transaction therefore the insert fails with the primary key violation.

              try to set your Isolation Level in the cache config, from your first post,

              FROM

              <attribute name="IsolationLevel">READ_COMMITTED</attribute>


              To

              <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
              
              OR
              
              <attribute name="IsolationLevel">SERIALIZABLE</attribute>
              
              


              Rerun you performance testing and let's know if you're still seeing the issue.

              • 4. Re: Problem with JBoss JDBCCacheLoader
                Archanaa Panda Newbie

                Thanks a lot, Hany. Yes I seem to have goofed up with the cache configuration. The real one is below :)). Sorry about that. I will definitely try your suggestion - this looks to be a very plausible reason for the problem. I will let you know the result in my next post. In the meantime, just take a look at the actual configuration I am posting below and let me know if you find anything else that attracts your attention-


                <?xml version="1.0" encoding="UTF-8"?>
                
                <!-- ===================================================================== -->
                <!-- -->
                <!-- Sample TreeCache Service Configuration -->
                <!-- -->
                <!-- ===================================================================== -->
                
                <server>
                
                 <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/>
                
                
                 <!-- ==================================================================== -->
                 <!-- Defines TreeCache configuration -->
                 <!-- ==================================================================== -->
                
                 <mbean code="org.jboss.cache.TreeCache"
                 name="jboss.cache:service=ArchTreeCache">
                
                 <depends>jboss:service=Naming</depends>
                 <depends>jboss:service=TransactionManager</depends>
                
                 <!--
                 Configure the TransactionManager
                 -->
                 <attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute>
                
                 <!--
                 Isolation level : SERIALIZABLE
                 REPEATABLE_READ (default)
                 READ_COMMITTED
                 READ_UNCOMMITTED
                 NONE
                 -->
                 <!--
                 <attribute name="NodeLockingScheme">OPTIMISTIC</attribute>
                 -->
                
                 <attribute name="IsolationLevel">READ_COMMITTED</attribute>
                
                 <!--
                 Valid modes are LOCAL
                 REPL_ASYNC
                 REPL_SYNC
                 INVALIDATION_ASYNC
                 INVALIDATION_SYNC
                 -->
                 <attribute name="CacheMode">REPL_SYNC</attribute>
                
                 <!--
                 Just used for async repl: use a replication queue
                 -->
                 <attribute name="UseReplQueue">false</attribute>
                
                 <!--
                 Replication interval for replication queue (in ms)
                 -->
                 <attribute name="ReplQueueInterval">0</attribute>
                
                 <!--
                 Max number of elements which trigger replication
                 -->
                 <attribute name="ReplQueueMaxElements">0</attribute>
                
                 <!-- Name of cluster. Needs to be the same for all clusters, in order
                 to find each other
                 -->
                 <attribute name="ClusterName">ArchTreeCache-Cluster</attribute>
                
                 <!-- JGroups protocol stack properties. Can also be a URL,
                 e.g. file:/home/bela/default.xml
                 <attribute name="ClusterProperties"></attribute>
                 -->
                
                 <attribute name="ClusterConfig">
                 <config>
                 <!-- UDP: if you have a multihomed machine,
                 set the bind_addr attribute to the appropriate NIC IP address, e.g bind_addr="192.168.0.2"
                 -->
                 <!-- UDP: On Windows machines, because of the media sense feature
                 being broken with multicast (even after disabling media sense)
                 set the loopback attribute to true -->
                 <UDP mcast_addr="228.1.3.4" mcast_port="48866"
                 ip_ttl="64" ip_mcast="true"
                 mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
                 ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
                 loopback="true"/>
                 <PING timeout="20000" num_initial_members="3"
                 up_thread="false" down_thread="false"/>
                 <MERGE2 min_interval="10000" max_interval="20000"/>
                 <!-- <FD shun="true" up_thread="true" down_thread="true" />-->
                 <FD_SOCK/>
                 <VERIFY_SUSPECT timeout="1500"
                 up_thread="false" down_thread="false"/>
                 <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
                 max_xmit_size="8192" up_thread="false" down_thread="false"/>
                 <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
                 down_thread="false"/>
                 <pbcast.STABLE desired_avg_gossip="20000"
                 up_thread="false" down_thread="false"/>
                 <FRAG frag_size="8192"
                 down_thread="false" up_thread="false"/>
                 <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
                 shun="true" print_local_addr="true"/>
                 <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
                 </config>
                 </attribute>
                
                
                 <!--
                 Whether or not to fetch state on joining a cluster
                 NOTE this used to be called FetchStateOnStartup and has been renamed to be more descriptive.
                 -->
                 <attribute name="FetchInMemoryState">true</attribute>
                
                 <!--
                 The max amount of time (in milliseconds) we wait until the
                 initial state (ie. the contents of the cache) are retrieved from
                 existing members in a clustered environment
                 -->
                 <attribute name="InitialStateRetrievalTimeout">500000</attribute>
                
                 <!--
                 Number of milliseconds to wait until all responses for a
                 synchronous call have been received.
                 -->
                 <attribute name="SyncReplTimeout">200000</attribute>
                
                 <!-- Max number of milliseconds to wait for a lock acquisition -->
                 <attribute name="LockAcquisitionTimeout">15000</attribute>
                
                
                
                 <!-- Name of the eviction policy class. -->
                 <attribute name="EvictionPolicyClass"></attribute>
                
                 <!--
                 Indicate whether to use marshalling or not. Set this to true if you are running under a scoped
                 class loader, e.g., inside an application server. Default is "false".
                 -->
                
                 <attribute name="CacheLoaderConfiguration">
                 <config>
                 <passivation>false</passivation>
                 <preload>/</preload>
                 <shared>true</shared>
                 <cacheloader>
                 <class>org.jboss.cache.loader.JDBCCacheLoader</class>
                 <properties>
                 cache.jdbc.table.name=jbosscache
                 cache.jdbc.table.create=true
                 cache.jdbc.table.drop=false
                 cache.jdbc.table.primarykey=jbosscache_pk
                 cache.jdbc.fqn.column=fqn
                 cache.jdbc.fqn.type=varchar(255)
                 cache.jdbc.node.column=node
                 cache.jdbc.node.type=blob
                 cache.jdbc.parent.column=parent
                 cache.jdbc.driver=oracle.jdbc.driver.OracleDriver
                 cache.jdbc.url=jdbc:oracle:thin:@132.186.106.26:1541:ACRMDEV
                 cache.jdbc.user=FASTDEV
                 cache.jdbc.password=fastdev123$
                 </properties>
                 <async>false</async>
                 <fetchPersistentState>true</fetchPersistentState>
                 <ignoreModifications>false</ignoreModifications>
                 </cacheloader>
                 </config>
                 </attribute>
                 </mbean>
                
                
                
                
                 <!-- Uncomment to get a graphical view of the TreeCache MBean above -->
                 <!-- <mbean code="org.jboss.cache.TreeCacheView" name="jboss.cache:service=TreeCacheView">-->
                 <!-- <depends>jboss.cache:service=TreeCache</depends>-->
                 <!-- <attribute name="CacheService">jboss.cache:service=TreeCache</attribute>-->
                 <!-- </mbean>-->
                
                
                </server>