Cache, missing entries

[baranowb]

I have encountered some problems with JBCache.
I have transactional enviroment. changes are commited to cache upon tx commit phase - classes that implement TransactionalAction are executed.

Hoever it misses some entries - cache doesnt retain entries it should upon concurrent tx running. Here is what is happening:

1)Tx[1] reads data to cache and stores to local Map
2)Tx[2] reads ......
3)Tx[1] adds "ALA"
4)Tx[1] adds "BEATA"
5)Tx[2] removes "OLA" - last 3 operations are performed on local copies each tx stores in Map

6)Tx[1] Commits - two new entries are added to Tx - they are present
7)Tx[2] commits - one entry is removed

Cache lacks "ALA" and "BEATA". In fact if our caching structure doesnt write all entries - to cache upon commit, those entries are missed too.

on steps 1-2 TreeCache.getKeys(nodeName) is called, and for each key value from that node is retrieved, retrieed values are stored in local Map - which is not shared among tx, each has its local map.
on steps 3-5 remove(key) and put(key,value) opearations are performed on local Maps - no cache operations are performed.
in those steps TransactionAction's are added to SynchronizationHandler lists are added.
On 6 tx.commit/TransactionManager.commit is called. actions stored in SynchHandler call TreeCache.put(nodeName,key,object) and remove(nodeName,key) (it could, if something was removed)
- putting again and removing undeaded entries - whole copy of map is put into cache again, and some entries are removed
On 7 is the same


On steps 6-7 if whole local Map is not put into cache it lacks those entries which has not been put again. I mean if only updated, added entries are put in cache, all other are not retained.
Is this expected behaviour? I have used jbcache 1.2 or 1.3 and newest version. Isolation level is READ COMMITED, with 1.4 I have tried Optimisitc locking scheme.

Any suggestions would be highly welcome.
Kind regards

[baranowb]

it should be TransactionalAction not TransactionAction

[brian.stansberry]

Questions:

1) Are Tx[1] and Tx[2] running in the same cache, or in 2 different caches that replicate?
2) Please describe in terms of JBC API and java.util.Map calls exactly what is happening in steps 1 - 5.

I expect Tx[1] and Tx[2] are not making modifications to the same underlying map.

[baranowb]

Both tx run on the same cache isntance.
Here is more datailed call flow

1-2) each tx has its own localCacheCopy

Set keys=treeCache.getKeys(nodeName)
Iterator it=keys.iterator()
Map localCacheCopy=....;
while(it.hastNext()
{
Object key=it.next();
Object val=treeCache.getObject("OURCACHE",key)
localCacheCopy.put(key,val)l
}

//Store localCacheCopy in tx local data.

3-4)
getLocalCacheCopy().put("ALA","ALA");
getLocalCacheCopy().put("BEATA","BEATA"); - retrieves localCacheCopy for Tx[1]
5)
getLocalCacheCopy().remove("OLA"); - retrieves localCacheCopy for Tx[2]

[brian.stansberry]

I don't see any puts or removes to the tree cache here. TreeCache is not aware of changes you make to localCacheCopy.

Also, getObject(String, Object) is not part of the TreeCache API. Did you mean "treeCache.get("OURCACHE", key)"?

[baranowb]

Yes I meant "get" not "getObject". YOu asked for desc of steps 1-5

6) here Tx commits, SynchronizationHanlder executes preCommitActions
Iterator iterc=getLocalCacheCopy().entrySet().iterator(); - Tx[1]
while (iterc.hasNext()) {
Map.Entry nextEntry = (Map.Entry)iterc.next();
cache.put(nodeName, nextEntry.getKey(), nextEntry.getValue());
}
//this is note executed since noetries have been removed
if(getRemovedEntriesSet().size()>0)
{
iterc=getRemovedEntriesSet().iterator();
while(iterc.hasNext())
{

Object key=iterc.next();
cache.remove(nodeName,key)

}
}


7) looks similar, however no put operation is performed, only one remove since entry has been removed and that key has been added to RemovedEntriesSet.

[brian.stansberry]

I suspect your problem arises from using a Synchronization to update JBC. JBC itself uses a Synchronization to control replication of transactional changes around the cluster. If your Synchronization's beforeCompletion is executed by the TransactionManager after JBC's, I'm not sure what will happen. Are you able to ensure that your Synchronization is registered with the transaction before any call goes into JBC?

I'm not sure from your original post -- do you see this with both pessimistic and optimistic locking?

[baranowb]

Do You have in mind something like that?
This is actual code snippet

SynchronizationHandler handler = new SynchronizationHandler(tx);

try {
tx.registerSynchronization(handler);
} catch (RollbackException e) {
// This would be extremely strange if this ever got called.
logger.error(e);
}


tx - retrieven from Transaction manager
SynchronizationHandler stores all action that are responsible for updating cache.

[brian.stansberry]

Yes -- does that tx.registerSynchronization() call always happen before any call is made to the cache with' tx' associated with the thread?

[baranowb]

Yes it happens right after tx is created, its done in the same function -
Thread calls begin - which creates Tx and registers SynchronizationHandler

[baranowb]

Hmm we could find solution so far, any ideas?

[brian.stansberry]

No really good ideas. One odd thing was in your steps 1 and 2:

Object val=treeCache.getObject("OURCACHE",key)

Should that be

Object val = treeCache.get(nodename, key)

You already confirmed it should be "get" not "getObject". But is the Fqn you are using the same as what you used in the treeCache.getKeys() call?

Also, does using REPEATABLE_READ help?

What you are doing is pretty similar to JBoss AS HttpSession replication with ATTRIBUTE granularity. Only difference is you are doing the updates to the cache via a Synchronization.

[baranowb]

By "our cache" I meant fqn of node.
yes I have tried RR, no improvement. I have created tck test which replicates problem, I can propably create standalone app which replicates it.
Tck test can be downloaded from here: https://mobicents.dev.java.net/issues/show_bug.cgi?id=139

[brian.stansberry]

If you can create a junit test case that recreates it, that would be best. If you can, please open a JIRA and attach it.