TreeCache CacheLoader Issue

[robnor]

Hi,

I have not used JBossCache before the project I am in, so I am green.

Background:
We have a application that we have split up into three parts; core, import and distribution. Each part is an ear, with ejb3's and a Jbosscache.

There is no problem at all in getting all caches to start and to function storing in memory data.

However. We need the cache to be persistent, thus I we have cacheloader, JDBC Cacheloader (to be more preicse).

I am using the TreeCache to store an object (ImportTracker), which holds data about the different imports. If the server somehow goes down, or if we drop an updated ear in the deploy directory, we want the application to be able to resume import, distribution etc where it was. Thus I need the in-memory cache to always write everything to the persistent store directly.

My issue is that regardless if I use filecacheloader or jdbccacheloader no data at all gets stored. If I activate debug level on jboss.cache, I can see that the transaction interceptor gets called but never the storage interceptor.

It is a bit unclear as to how the treecache is suppoed to work. If I create a node say: import/host1/

The map object has two entries, tracker and distribution

Tracker is an object with other object inside each describing a import, and distribution is also a object hoding info about different distributions.

Question: After creation of node and put of object (something like) cache.put("/import/host1", "tracker", myobject);

Later in the app when I retrieve the object, and make changes to it. Does the changes automatically update cache? or do I need to make a put everytime (to trigger update).

Anyways, my issue is that I cannot get the cacheloader to store the data. The only time I can get it to work as I want is when I retrieve the cache, and from the cahce retrieve the cacheloader and then make a put directly on that. This cannot be the way to go?

Please advice :-)

[robnor]

Log:

2007-05-08 11:33:40,093 DEBUG [ImportCacheHelper] storeTracker() - start
2007-05-08 11:33:40,093 DEBUG [ImportCacheHelper] ImportTracker - PUT
2007-05-08 11:33:40,093 DEBUG [ImportCacheHelper] TRANSACTION STATUS: Active [0]
2007-05-08 11:33:40,093 DEBUG [TxInterceptor] local transaction exists - registering global tx if not present for Thread[WorkManager(3)-49,5,JBoss Pooled Threads]
2007-05-08 11:33:40,093 DEBUG [TxInterceptor] Transaction TransactionImpl:XidImpl[FormatId=257, GlobalId=dev-07/498, BranchQual=, localId=498] is already registered.
2007-05-08 11:33:40,109 DEBUG [TxInterceptor] local transaction exists - registering global tx if not present for Thread[WorkManager(3)-49,5,JBoss Pooled Threads]
2007-05-08 11:33:40,109 DEBUG [TxInterceptor] Transaction TransactionImpl:XidImpl[FormatId=257, GlobalId=dev-07/498, BranchQual=, localId=498] is already registered.
2007-05-08 11:33:40,109 DEBUG [ImportCacheHelper] PUT - Performed!
2007-05-08 11:33:40,109 DEBUG [ImportCacheHelper] storeTracker() - stop
2007-05-08 11:33:40,109 DEBUG [LocalScpPusher1] PUSHED PRODUCT: BatchId[1303016476664275:061432324027] ProductType[DummyFilterProductsNoDrm] Pusher[LocalSCPPusher1]
2007-05-08 11:33:40,125 DEBUG [TxInterceptor] local transaction exists - registering global tx if not present for Thread[WorkManager(3)-49,5,JBoss Pooled Threads]
2007-05-08 11:33:40,125 DEBUG [TxInterceptor] Transaction TransactionImpl:XidImpl[FormatId=257, GlobalId=dev-07/498, BranchQual=, localId=498] is already registered.
2007-05-08 11:33:40,859 DEBUG [TxInterceptor] Running commit phase. One phase? false
2007-05-08 11:33:40,859 DEBUG [OptimisticValidatorInterceptor] commiting validated changes
2007-05-08 11:33:40,859 DEBUG [TxInterceptor] Finished local commit/rollback method for GlobalTransaction:<null>:76
2007-05-08 11:33:40,859 DEBUG [TxInterceptor] Finished commit phase
2007-05-08 11:33:40,875 DEBUG [TxInterceptor] local transaction exists - registering global tx if not present for Thread[WorkManager(3)-50,5,JBoss Pooled Threads]
2007-05-08 11:33:40,875 DEBUG [TxInterceptor] Running commit phase. One phase? false
2007-05-08 11:33:40,875 DEBUG [OptimisticValidatorInterceptor] commiting validated changes
2007-05-08 11:33:40,875 DEBUG [TxInterceptor] Finished local commit/rollback method for GlobalTransaction:<null>:77
2007-05-08 11:33:40,875 DEBUG [TxInterceptor] Finished commit phase
2007-05-08 11:33:40,890 DEBUG [ImportCacheHelper] storeTracker() - start
2007-05-08 11:33:40,890 DEBUG [ImportCacheHelper] ImportTracker - PUT
2007-05-08 11:33:40,890 DEBUG [ImportCacheHelper] NO TRANSACTION
2007-05-08 11:33:40,890 DEBUG [TxInterceptor] local transaction exists - registering global tx if not present for Thread[WorkManager(3)-50,5,JBoss Pooled Threads]
2007-05-08 11:33:40,890 DEBUG [TxInterceptor] Running commit phase. One phase? false
2007-05-08 11:33:40,890 DEBUG [OptimisticValidatorInterceptor] commiting validated changes
2007-05-08 11:33:40,890 DEBUG [TxInterceptor] Finished local commit/rollback method for GlobalTransaction:<null>:78
2007-05-08 11:33:40,890 DEBUG [TxInterceptor] Finished commit phase
2007-05-08 11:33:40,890 DEBUG [TxInterceptor] local transaction exists - registering global tx if not present for Thread[WorkManager(3)-50,5,JBoss Pooled Threads]
2007-05-08 11:33:40,890 DEBUG [TxInterceptor] Running commit phase. One phase? false
2007-05-08 11:33:40,906 DEBUG [OptimisticValidatorInterceptor] commiting validated changes
2007-05-08 11:33:40,906 DEBUG [TxInterceptor] Finished local commit/rollback method for GlobalTransaction:<null>:79
2007-05-08 11:33:40,906 DEBUG [TxInterceptor] Finished commit phase
2007-05-08 11:33:40,906 DEBUG [ImportCacheHelper] PUT - Performed!

The storage interceptor is never triggered and the data is not commited in the cacheloader.

I can get the data to get stored as I want it to be but then I need to get the cacheloader from the cache and make a put directly in the cacheloader. This sux!

[robnor]

Further

Cache config:

 <mbean code="org.jboss.cache.TreeCache" name="ice.import:service=TreeCache">
 <depends>jboss:service=Naming</depends>
 <depends>jboss:service=TransactionManager</depends>
 <depends>jboss.jca:service=DataSourceBinding,name=DefaultDS</depends> _


 <!-- Configure the TransactionManager -->
 <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>
 <attribute name="NodeLockingScheme">OPTIMISTIC</attribute>
 <attribute name="IsolationLevel">REPEATABLE_READ</attribute>
<!-- <attribute name="IsolationLevel">NONE</attribute>-->


 <attribute name="CacheMode">LOCAL</attribute>
<!-- <attribute name="CacheMode">REPL_SYNC</attribute>-->
<!-- <attribute name="ClusterName">Import-Cache-Cluster</attribute>-->

 <!-- Configure Marshalling -->
 <attribute name="InitialStateRetrievalTimeout">5000</attribute>
 <attribute name="SyncReplTimeout">10000</attribute>
 <attribute name="LockAcquisitionTimeout">15000</attribute>

 <attribute name="CacheLoaderConfiguration" replace="false">

 <config>
 <preload>/import</preload>
 <shared>false</shared>

 <cacheloader>
 <class>org.jboss.cache.loader.JDBCCacheLoader</class>
 <properties>
 cache.jdbc.datasource=java:/DefaultDS
 cache.jdbc.table.drop=false
 </properties>
 <async>false</async>
 <fetchPersistentState>true</fetchPersistentState>
 <ignoreModifications>false</ignoreModifications>
 <purgeOnStartup>false</purgeOnStartup>
 </cacheloader>

 <cacheloader>
 <class>org.jboss.cache.loader.FileCacheLoader</class>
 <properties>location=/tmp</properties>
 <async>false</async>
 <fetchPersistentState>false</fetchPersistentState>
 <ignoreModifications>false</ignoreModifications>
 <purgeOnStartup>false</purgeOnStartup>
 </cacheloader>


 </config>
 </attribute>

 </mbean>

I have been experimenting with different configuration without success, latest try is with optimistic node locking...

I have the extra file cacheloader to be able to view what is stored without having to redeploy app (to see if it fetches the state when it loads).

[robnor]

My current workaround:

 public static void storeTracker(ImportTracker tracker)
 {
 logDebug("storeTracker() - start");

 try
 {
 if (useTreeCache)
 {
 logDebug("ImportTracker - PUT");
 boolean transactionActive = false;
 try
 {
 TreeCacheMBean cache = CacheHelper.getImportCache();
 Transaction trans = CacheHelper.getImportCache().getTransactionManager().getTransaction();
 if (trans != null)
 {
 logDebug("TRANSACTION STATUS: " + getTransactionStatusString(trans.getStatus()));
 if (trans.getStatus() != Status.STATUS_COMMITTED)
 {
 transactionActive = true;
 }
 }
 else
 {
 logDebug("NO TRANSACTION");
 transactionActive = false;
 }
 }
 catch (SystemException ex)
 {
 slog.error("Transaction Error!", ex);
 }

 CacheHelper.getImportCache().get(nodename).put("importTracker", tracker);


 Fqn fqn = CacheHelper.getImportCache().get(nodename).getFqn();
 try
 {
 ImportTracker tr = (ImportTracker) CacheHelper.getImportCache().get(fqn, "importTracker");
 ImportTracker tr2 = (ImportTracker) CacheHelper.getImportCache().getCacheLoader().get(fqn).get("importTracker");
 if (tracker.equals(tr))
 {
 logDebug("Tracker objects are EQUAL");
 }
 else
 {
 logDebug("Tracker object are NOT EQUAL");
 logDebug("Putting updated Tracker hard in cacheloader");
 if (!transactionActive)
 {
 CacheHelper.getImportCache().getInstance().getCacheLoader().put(fqn, "importTracker", tracker);
 logDebug("Done!");
 }
 }
 logDebug("COMPARING WITH Tracker received from cacheloader");
 if (tracker.equals(tr2))
 {
 logDebug("Tracker objects are EQUAL");
 }
 else
 {
 logDebug("Tracker object are NOT EQUAL");
 logDebug("Putting updated Tracker hard in cacheloader");
 if (!transactionActive)
 {
 CacheHelper.getImportCache().getInstance().getCacheLoader().put(fqn, "importTracker", tracker);
 logDebug("Done!");
 }
 }


 }
 catch (Exception ex)
 {
 ex.printStackTrace();
 }
 logDebug("PUT - Performed!");
 }
 else
 {
 // === this should not be necessary as aop detects setting on methods ===
 CacheHelper.getImportPojoCache().putObject(nodename, tracker);
 }
 }
 catch (CacheException ex)
 {
 slog.error("Failed to store importtracker in cache", ex);
 }
 logDebug("storeTracker() - stop");

 }


 public static ImportTracker getTracker()
 {
 ImportTracker retval = null;
 if (useTreeCache)
 {
 TreeCacheMBean cache = CacheHelper.getImportCache();

 if (!cache.exists(nodename, "importTracker"))
 {
 logDebug(nodename + "/importTracker does not exist, creating a new");
 ImportTracker tracker = new ImportTracker();
 try
 {
 cache.put(nodename, "importTracker", tracker);
 retval = (ImportTracker) cache.get(nodename).get("importTracker");
 }
 catch (CacheException e)
 {
 slog.error("Failed to retrieve ImportTracker", e);
 }
 }
 else
 {
 try
 {
 retval = (ImportTracker) cache.get(nodename).get("importTracker");
 }
 catch (CacheException e)
 {
 slog.error(e);
 }
 }
 }
 else
 {
 PojoCacheMBean cache = CacheHelper.getImportPojoCache();

 if (!cache.exists(nodename, "importTracker"))
 {
 slog.debug(nodename + "/importTracker does not exist, creating a new");
 ImportTracker tracker = new ImportTracker();
 try
 {
 cache.putObject(nodename, tracker);
 retval = (ImportTracker) cache.getObject(nodename);
 }
 catch (CacheException e)
 {
 slog.error("Failed to create ImportTracker", e);
 }
 }
 else
 {
 try
 {
 retval = (ImportTracker) cache.getObject(nodename);
 }
 catch (CacheException e)
 {
 slog.error(e);
 }
 }
 }
 return retval;
 }

plese dont look at the pojocahce stuff, it will not be used atm

[robnor]

Hmm.. no replies :-)

I suppose that this could be a bug in jboss as aop implementation that does not trigger interceptor as it should.

It could also be that treecache bug where the pointcut definition is not correct.

Then of course it could be me that has used treecache in the wrong way... (Or the product is insufficiently documented...)

:-)

So I guess I have to run with my workaround...

[genman]

I didn't really dig into your specific issue, but I do have some general advice.

Regarding persistence, there are some bundled configurations that use cache loaders. You can try to use these as a "sanity check". These configurations are unit tested and should work. Check out the unit tests for how they work with transactions as well...

[robnor]

I used fisheeye recommended-config.xml and modified it for my purposes.

I have checked out alternative configurations with evictionrules etc, but they still does not trigger the store interceptor, thus leaving the persistant state and the in-memory state inconsistent with eachother.

I am still forced to getcache, and from the cache get cacheloader and do put there, to make it work. Crazy...

If I have to do that I could almost just as well just store a blob in database and update it there... hehe

[brian.stansberry]

Here's your problem:

CacheHelper.getImportCache().get(nodename).put("importTracker", tracker);

Use:

CacheHelper.getImportCache().put(nodename, "importTracker", tracker);

Do the same anywhere else you have the same kind of thing going on.

The 1.x "public Node get(Fqn)" method was a bad idea and is conceptually being completely reworked in JBC 2.0. If you invoke directly on a JBC 1.x Node rather than through the cache, your call bypasses all the interceptors that do stuff like cache loader integration, replication, locking, etc.

To answer your question about having to call put() again if you modify your object, yes you do, unless you prepare your object for JBoss Dynamic AOP and use PojoCache.

[robnor]

I tried to change this.

I now have major problem with node locking errors.

It seems it cannot cope with highfrequency updates?

I have an Ojbect (ImportTracker) that holds multiple objects (BatchTracker). The ImportTracker holds all the BatchTrackers, the BatchTracker holds data about the import of a specific Batch.

Further I have a DistributionManager. The Batch is productified and distributed to several hubs. The distributionManager is the one responsible for keeping track of the distribution.

A Batch, can be productified into several different products. There are multiple hubs requesting different kinds of products. The Distributionmanager keeps track that the Batch and its products are being delivered to the hubs requesting the producttype. The batchimport is not done until all producttypes has been delivered to all hubs.

Then we may have say 3 machines dedicated to import all working on a local cache which is persisted in a single database (to avoid overwrite they store objects in different nodes for example /import/<host1>/importTracker).

This will make the application to make many many put to the persistent storage.

Can get around nodelocking? or perhaps jbosscache is not the tool I should use?

Anyway, your comment solved the issue regarding not "flushing" the inmemory data to persistent storage.

[robnor]

Thank you! :-)

[manik]

Node locking and concurrency is important to retain data integrity. Do you have a high percentage of writes? If so, then a caching library is NOT what you need. (think about it! :)

If you want to use JBoss Cache as a distributed in-memory database, then I'd recommend using pessimistic locking and a high lock acquisition timeout, but expect transactions to block considerably.