Problem with JBoss JDBCCacheLoader

[arcpanda]

Hello,

We are using JBoss Cache 1.3 for our project which is under development. Recently while doing some performance testing we found a problem with the JDBC Cache Loader (we are using an Oracle database as the backing store for our JBoss Cache). When we run with multiple threads or sometimes even one thread, we get the following exception:

java.lang.RuntimeException: java.lang.IllegalStateException: Failed to insert node: ORA-00001: unique constraint (FASTDEV.JBOSSCACHE_PK) violated at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:4810)
at org.jboss.cache.TreeCache.put(TreeCache.java:3242)
at com.sisl.ashleycomm.cache.CacheTimeoutManager.putCacheable(CacheTimeoutManager.java:452)
at com.sisl.ashleycomm.cache.push.command.MobilePushCacheTimeoutManager.putCacheable(MobilePushCacheTimeoutManager.java:168)
at com.sisl.ashleycomm.cache.DefaultCacheableVersionProxy.processEvent(DefaultCacheableVersionProxy.java:134)
at com.sisl.ashleycomm.cache.push.command.MobilePushCacheTimeoutManager.getMobilePushCommandsForGprsSend


Caused by: java.lang.IllegalStateException: Failed to insert node: ORA-00001: unique constraint (FASTDEV.JBOSSCACHE_PK) violated
at org.jboss.cache.loader.JDBCCacheLoader.insertNode(JDBCCacheLoader.java:982)
at org.jboss.cache.loader.JDBCCacheLoader.put(JDBCCacheLoader.java:325)
at org.jboss.cache.interceptors.CacheStoreInterceptor.invoke(CacheStoreInterceptor.java:202)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
at org.jboss.cache.interceptors.CacheLoaderInterceptor.invoke(CacheLoaderInterceptor.java:187)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:32)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:328)
at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:139)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:153)
at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:4804)


Actually the scenario is like this:
A thread gets a node on the cache and then does some modification on the node and tries to put it back on the cache.
There may be multiple threads accessing the same node.

Looking at the source code of JBoss Cache 1.3, I find that when put() is called on JDBCCacheLoader, loadNode() method is called. Apparantly this is returning null so the insertNode() is being called. But then when it tries to do the insertNode, it gets the PK violated error which indicates the row was existing in the database already!!!

What are the precautions we need to take to avoid this problem? Is our setting wrong?

A quick reply would be very much appreciated. Please let me know if there are more details required for diagnosing the problem

Thanks and Regards,
Archanaa Panda

Following are the details of our cache config xml file-

<attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute>
 <attribute name="IsolationLevel">READ_COMMITTED</attribute>
<attribute name="CacheMode">REPL_SYNC</attribute>


<attribute name="CacheLoaderConfiguration">
 <config>
 <passivation>false</passivation>
 <preload>/</preload>
 <shared>true</shared>
 <cacheloader>
 <class>org.jboss.cache.loader.JDBCCacheLoader</class>
 <properties>
 cache.jdbc.table.name=jbosscache
 cache.jdbc.table.create=true
 cache.jdbc.table.drop=false
 cache.jdbc.table.primarykey=jbosscache_pk
 cache.jdbc.fqn.column=fqn
 cache.jdbc.fqn.type=varchar(255)
 cache.jdbc.node.column=node
 cache.jdbc.node.type=blob
 cache.jdbc.parent.column=parent
 cache.jdbc.driver=oracle.jdbc.driver.OracleDriver
 cache.jdbc.url=jdbc:oracle:thin:@132.186.106.26:1541:ACRMDEV
 cache.jdbc.user=FASTDEV
 cache.jdbc.password=fastdev123$
 </properties>
 <async>false</async>
 <fetchPersistentState>true</fetchPersistentState>
 <ignoreModifications>false</ignoreModifications>
 </cacheloader>
 </config>
 </attribute>

[hmesha]

I think you have a race condition going on here stems from your application doing this call com.sisl.ashleycomm.cache.CacheTimeoutManager.putCacheable(CacheTimeoutManager.java:452)

Are you using eviction policy? Can you post the entire JBossCache configuration you're using?

what isolation level you're using?

[arcpanda]

Thank you so very much for looking into the problem. I was on leave as i had fallen ill and i just saw your reply. Our isolation level is REPEATABLE_READ and we are using REPL_SYNC cache mode

Please find below the complete configuration-

<?xml version="1.0" encoding="UTF-8"?>

<!-- ===================================================================== -->
<!-- -->
<!-- Sample TreeCache Service Configuration -->
<!-- -->
<!-- ===================================================================== -->

<server>

 <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/>


 <!-- ==================================================================== -->
 <!-- Defines TreeCache configuration -->
 <!-- ==================================================================== -->

 <mbean code="org.jboss.cache.TreeCache"
 name="jboss.cache:service=TreeCache">

 <depends>jboss:service=Naming</depends>
 <depends>jboss:service=TransactionManager</depends>

 <!--
 Configure the TransactionManager
 -->
 <attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute>

 <!--
 Isolation level : SERIALIZABLE
 REPEATABLE_READ (default)
 READ_COMMITTED
 READ_UNCOMMITTED
 NONE
 -->
 <!--
 <attribute name="NodeLockingScheme">OPTIMISTIC</attribute>
 -->
 <attribute name="IsolationLevel">REPEATABLE_READ</attribute>

 <!--
 Valid modes are LOCAL
 REPL_ASYNC
 REPL_SYNC
 INVALIDATION_ASYNC
 INVALIDATION_SYNC
 -->
 <attribute name="CacheMode">REPL_SYNC</attribute>

 <!--
 Just used for async repl: use a replication queue
 -->
 <attribute name="UseReplQueue">false</attribute>

 <!--
 Replication interval for replication queue (in ms)
 -->
 <attribute name="ReplQueueInterval">0</attribute>

 <!--
 Max number of elements which trigger replication
 -->
 <attribute name="ReplQueueMaxElements">0</attribute>

 <!-- Name of cluster. Needs to be the same for all clusters, in order
 to find each other
 -->
 <attribute name="ClusterName">TreeCache-Cluster</attribute>

 <!-- JGroups protocol stack properties. Can also be a URL,
 e.g. file:/home/bela/default.xml
 <attribute name="ClusterProperties"></attribute>
 -->

 <attribute name="ClusterConfig">
 <config>
 <!-- UDP: if you have a multihomed machine,
 set the bind_addr attribute to the appropriate NIC IP address, e.g bind_addr="192.168.0.2"
 -->
 <!-- UDP: On Windows machines, because of the media sense feature
 being broken with multicast (even after disabling media sense)
 set the loopback attribute to true -->
 <UDP mcast_addr="228.1.2.3" mcast_port="48866"
 ip_ttl="64" ip_mcast="true"
 mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
 ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
 loopback="false"/>
 <PING timeout="2000" num_initial_members="3"
 up_thread="false" down_thread="false"/>
 <MERGE2 min_interval="10000" max_interval="20000"/>
 <!-- <FD shun="true" up_thread="true" down_thread="true" />-->
 <FD_SOCK/>
 <VERIFY_SUSPECT timeout="1500"
 up_thread="false" down_thread="false"/>
 <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
 max_xmit_size="8192" up_thread="false" down_thread="false"/>
 <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
 down_thread="false"/>
 <pbcast.STABLE desired_avg_gossip="20000"
 up_thread="false" down_thread="false"/>
 <FRAG frag_size="8192"
 down_thread="false" up_thread="false"/>
 <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
 shun="true" print_local_addr="true"/>
 <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
 </config>
 </attribute>


 <!--
 Whether or not to fetch state on joining a cluster
 NOTE this used to be called FetchStateOnStartup and has been renamed to be more descriptive.
 -->
 <attribute name="FetchInMemoryState">true</attribute>

 <!--
 The max amount of time (in milliseconds) we wait until the
 initial state (ie. the contents of the cache) are retrieved from
 existing members in a clustered environment
 -->
 <attribute name="InitialStateRetrievalTimeout">15000</attribute>

 <!--
 Number of milliseconds to wait until all responses for a
 synchronous call have been received.
 -->
 <attribute name="SyncReplTimeout">15000</attribute>

 <!-- Max number of milliseconds to wait for a lock acquisition -->
 <attribute name="LockAcquisitionTimeout">10000</attribute>

 <!-- Name of the eviction policy class. -->
 <attribute name="EvictionPolicyClass"></attribute>

 <!--
 Indicate whether to use marshalling or not. Set this to true if you are running under a scoped
 class loader, e.g., inside an application server. Default is "false".
 -->
 <attribute name="UseMarshalling">false</attribute>
 </mbean>


 <!-- Uncomment to get a graphical view of the TreeCache MBean above -->
 <!-- <mbean code="org.jboss.cache.TreeCacheView" name="jboss.cache:service=TreeCacheView">-->
 <!-- <depends>jboss.cache:service=TreeCache</depends>-->
 <!-- <attribute name="CacheService">jboss.cache:service=TreeCache</attribute>-->
 <!-- </mbean>-->


</server>

[hmesha]

Are you sure this's your current cache configuration? It doesn't have even the JDBC cache loader configuration which was in your first post?!!

Anyway, I think based on the your first post cache configuration that more than one thread is accessing the data with read committed isolation level which means that Non-repeatable reads may occur. That would result in data retrieved in a SELECT statement may be modified by some other transaction when it commits.

Actually the scenario is like this:
A thread gets a node on the cache and then does some modification on the node and tries to put it back on the cache.
There may be multiple threads accessing the same node.

So when you do a get on the node (SELECT) the entry wasn't there and therefore loadNode() returns null but then when the transcation commit to the database trying to put the node (INSERT) rather than (UPDATE) the entry was already added by another transaction therefore the insert fails with the primary key violation.

try to set your Isolation Level in the cache config, from your first post,

FROM

<attribute name="IsolationLevel">READ_COMMITTED</attribute>

To

<attribute name="IsolationLevel">REPEATABLE_READ</attribute>

OR

<attribute name="IsolationLevel">SERIALIZABLE</attribute>

Rerun you performance testing and let's know if you're still seeing the issue.

[arcpanda]

Thanks a lot, Hany. Yes I seem to have goofed up with the cache configuration. The real one is below :)). Sorry about that. I will definitely try your suggestion - this looks to be a very plausible reason for the problem. I will let you know the result in my next post. In the meantime, just take a look at the actual configuration I am posting below and let me know if you find anything else that attracts your attention-

<?xml version="1.0" encoding="UTF-8"?>

<!-- ===================================================================== -->
<!-- -->
<!-- Sample TreeCache Service Configuration -->
<!-- -->
<!-- ===================================================================== -->

<server>

 <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar"/>


 <!-- ==================================================================== -->
 <!-- Defines TreeCache configuration -->
 <!-- ==================================================================== -->

 <mbean code="org.jboss.cache.TreeCache"
 name="jboss.cache:service=ArchTreeCache">

 <depends>jboss:service=Naming</depends>
 <depends>jboss:service=TransactionManager</depends>

 <!--
 Configure the TransactionManager
 -->
 <attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute>

 <!--
 Isolation level : SERIALIZABLE
 REPEATABLE_READ (default)
 READ_COMMITTED
 READ_UNCOMMITTED
 NONE
 -->
 <!--
 <attribute name="NodeLockingScheme">OPTIMISTIC</attribute>
 -->

 <attribute name="IsolationLevel">READ_COMMITTED</attribute>

 <!--
 Valid modes are LOCAL
 REPL_ASYNC
 REPL_SYNC
 INVALIDATION_ASYNC
 INVALIDATION_SYNC
 -->
 <attribute name="CacheMode">REPL_SYNC</attribute>

 <!--
 Just used for async repl: use a replication queue
 -->
 <attribute name="UseReplQueue">false</attribute>

 <!--
 Replication interval for replication queue (in ms)
 -->
 <attribute name="ReplQueueInterval">0</attribute>

 <!--
 Max number of elements which trigger replication
 -->
 <attribute name="ReplQueueMaxElements">0</attribute>

 <!-- Name of cluster. Needs to be the same for all clusters, in order
 to find each other
 -->
 <attribute name="ClusterName">ArchTreeCache-Cluster</attribute>

 <!-- JGroups protocol stack properties. Can also be a URL,
 e.g. file:/home/bela/default.xml
 <attribute name="ClusterProperties"></attribute>
 -->

 <attribute name="ClusterConfig">
 <config>
 <!-- UDP: if you have a multihomed machine,
 set the bind_addr attribute to the appropriate NIC IP address, e.g bind_addr="192.168.0.2"
 -->
 <!-- UDP: On Windows machines, because of the media sense feature
 being broken with multicast (even after disabling media sense)
 set the loopback attribute to true -->
 <UDP mcast_addr="228.1.3.4" mcast_port="48866"
 ip_ttl="64" ip_mcast="true"
 mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
 ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
 loopback="true"/>
 <PING timeout="20000" num_initial_members="3"
 up_thread="false" down_thread="false"/>
 <MERGE2 min_interval="10000" max_interval="20000"/>
 <!-- <FD shun="true" up_thread="true" down_thread="true" />-->
 <FD_SOCK/>
 <VERIFY_SUSPECT timeout="1500"
 up_thread="false" down_thread="false"/>
 <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
 max_xmit_size="8192" up_thread="false" down_thread="false"/>
 <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10"
 down_thread="false"/>
 <pbcast.STABLE desired_avg_gossip="20000"
 up_thread="false" down_thread="false"/>
 <FRAG frag_size="8192"
 down_thread="false" up_thread="false"/>
 <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
 shun="true" print_local_addr="true"/>
 <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
 </config>
 </attribute>


 <!--
 Whether or not to fetch state on joining a cluster
 NOTE this used to be called FetchStateOnStartup and has been renamed to be more descriptive.
 -->
 <attribute name="FetchInMemoryState">true</attribute>

 <!--
 The max amount of time (in milliseconds) we wait until the
 initial state (ie. the contents of the cache) are retrieved from
 existing members in a clustered environment
 -->
 <attribute name="InitialStateRetrievalTimeout">500000</attribute>

 <!--
 Number of milliseconds to wait until all responses for a
 synchronous call have been received.
 -->
 <attribute name="SyncReplTimeout">200000</attribute>

 <!-- Max number of milliseconds to wait for a lock acquisition -->
 <attribute name="LockAcquisitionTimeout">15000</attribute>



 <!-- Name of the eviction policy class. -->
 <attribute name="EvictionPolicyClass"></attribute>

 <!--
 Indicate whether to use marshalling or not. Set this to true if you are running under a scoped
 class loader, e.g., inside an application server. Default is "false".
 -->

 <attribute name="CacheLoaderConfiguration">
 <config>
 <passivation>false</passivation>
 <preload>/</preload>
 <shared>true</shared>
 <cacheloader>
 <class>org.jboss.cache.loader.JDBCCacheLoader</class>
 <properties>
 cache.jdbc.table.name=jbosscache
 cache.jdbc.table.create=true
 cache.jdbc.table.drop=false
 cache.jdbc.table.primarykey=jbosscache_pk
 cache.jdbc.fqn.column=fqn
 cache.jdbc.fqn.type=varchar(255)
 cache.jdbc.node.column=node
 cache.jdbc.node.type=blob
 cache.jdbc.parent.column=parent
 cache.jdbc.driver=oracle.jdbc.driver.OracleDriver
 cache.jdbc.url=jdbc:oracle:thin:@132.186.106.26:1541:ACRMDEV
 cache.jdbc.user=FASTDEV
 cache.jdbc.password=fastdev123$
 </properties>
 <async>false</async>
 <fetchPersistentState>true</fetchPersistentState>
 <ignoreModifications>false</ignoreModifications>
 </cacheloader>
 </config>
 </attribute>
 </mbean>




 <!-- Uncomment to get a graphical view of the TreeCache MBean above -->
 <!-- <mbean code="org.jboss.cache.TreeCacheView" name="jboss.cache:service=TreeCacheView">-->
 <!-- <depends>jboss.cache:service=TreeCache</depends>-->
 <!-- <attribute name="CacheService">jboss.cache:service=TreeCache</attribute>-->
 <!-- </mbean>-->


</server>