JBossDeveloper

Never mind I found it, man I'm always doing that, I search for an hour then I ask the question, then I find it myself.

So embarrassing

Hi there

Have you had a chance to give this a go yet? I'd be keen to know what people think of Optimistic Locking, both in terms of what it does, performance, and stability.

Thanks,
Manik

"DashV" wrote:
Heya does the Optimistic Locking stuff work when using JBossCache as an MBean within JBoss 4.0.2.

I tried dropping jboss-cache.jar into my lib directory (overwriting the old one) and adding the attribute:

<attribute name="NodeLockingScheme">OPTIMISTIC</attribute>

to my caches service.xml. Now I get the following exception:

org.jboss.deployment.DeploymentException: No Attribute found with name: NodeLockingScheme
 at org.jboss.system.ServiceConfigurator.configure(ServiceConfigurator.java:263)
 at org.jboss.system.ServiceConfigurator.internalInstall(ServiceConfigurator.java:164)
 at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:118)
 at org.jboss.system.ServiceController.install(ServiceController.java:202)
 at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
 at $Proxy4.install(Unknown Source)

I was hoping to test and re-bench our JBoss 4.0.2 cluster configuration here as it uses 4 seperate jboss caches (three of them replicate across the cluster and one of those three is ASYNC.) So its a pretty good test of most of the the jboss cache fuctionality.

Unfortunately due to the CacheLoader startup also being busted I haven't been able to do any testing of 1.2.4beta with jboss4.0.2. I tried moving the code from createService back to startService as noted in this jira: http://jira.jboss.com/jira/browse/JBCACHE-303

Then I started getting NPEs on startup. I was going to look into it further until we discovered a data corruption problem that effects JBossCache 1.2.3 and 1.2.4beta (and likely most older releases) when using MySQLs driver for the JDBC CacheLoader. (we tried both their latest 3.0.x and 3.1.x connectors)

The problem really isn't with your code. It's with MySQL not properly implementing the methods: .getBinaryStream and .setBinaryStream.

The problem with these methods in the MySQL connector is that they only stream the byte array up to about 4k. So on insert anything longer than 4k gets truncated to 4k and inserted (your insertNode and updateNode are effected) without any warning or exception!!! This in turn corrupts the entire table (and our cache) and causes .getBinaryStream calls (in your case loadNode) to get corrupted garbage!!

The worst part is MySQL has had postings and complaints about this issue for at least 3 years and never fixed it. (The oracle driver had the same exact issue with these methods for two years as well. I haven't used Oracle in the past year so they may have fixed it. But I doubt it.)

Of course you could write your own streaming code but when you get into writing your own JDBC totally falls apart as you are then subjected to each vendors specific implementation classes for Blob. (if they even have such a thing) :P

In our usage we have two cache loaders one that uses Hypersonic and one that uses MySQL. Luckily we did find a fix that works for both but I'm pretty sure it won't work for Oracle. Our fix was to use .setObject and .getObject. These stream perfectly for the MySQL and Hypersonic drivers! (but it did not properly stream with Oracle when we tried it a year ago)

Also For Hypersonic it requires you set cache.jdbc.node.type=OBJECT
in your cacheloader configuration.

I can email you our modified version of JDBCCache loader if you guys want to test it in oracle... Maybe it works now.

If not then I really am at a loss for how to fix this and strick to straight JDBC as every vendor seems hell bent on claiming JDBC compliance and then leaves critical methods like these either silently busted or unimplemented entirely.

Also where is the jbosscache 1.2.3 tag in cvs? I found a tag for every version except 1.2.3 (we're pointed at your new cvs server) so we had to apply the fix to the HEAD then grab the compiled JDBCCacheLoader class and shove it into the 1.2.3 jar. Anyway it works now for us :) Wish I had a universal fix to give ya.

Hopefully this has been mostly informative and minimal rant. ;)

Here is are the tree snippets that make up the fix we came up with yesterday for our production build.

For the method:
private void insertNode(Fqn name, Map node)

 if(node != null)
 {
// Object marshalledNode = new MarshalledValue(node);
// ByteArrayOutputStream baos = new ByteArrayOutputStream();
// ObjectOutputStream oos = new ObjectOutputStream(baos);
// oos.writeObject(marshalledNode);
//
// ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
// ps.setBinaryStream(2, bais, baos.size());
 ps.setObject(2, new MarshalledValue(node));
 }
 else
 {
 ps.setNull(2, Types.BLOB);
 //ps.setNull(2, Types.LONGVARBINARY);
 }

 if(rs.next())
 {
 rowExists = true;
 //InputStream is = rs.getBinaryStream(1);
 Object marshalledNode = rs.getObject(1);

 //if(is != null && !rs.wasNull())
 if(marshalledNode != null)
 {
 //ObjectInputStream ois = null;
 try
 {
 // deserialize result
 //ois = new ObjectInputStream(is);
 //Object marshalledNode = ois.readObject();

 // de-marshall value if possible
 if(marshalledNode instanceof MarshalledValue)
 {
 oldNode = (Map) ((MarshalledValue) marshalledNode).get();
 }
 else if(marshalledNode instanceof MarshalledObject)
 {
 oldNode = (Map) ((MarshalledObject) marshalledNode).get();
 }
 }
 catch(IOException e)
 {
 throw new SQLException("Unable to load to deserialize result: " + e);
 }
 catch(ClassNotFoundException e)
 {
 throw new SQLException("Unable to load to deserialize result: " + e);
 }
// finally
// {
// safeClose(ois);
// }
 }

 if(node == null)
 {
 ps.setNull(1, Types.BLOB);
 //ps.setNull(1, Types.LONGVARBINARY);
 }
 else
 {
 //Object marshalledNode = new MarshalledValue(node);
 //ByteArrayOutputStream baos = new ByteArrayOutputStream();
 //ObjectOutputStream oos = new ObjectOutputStream(baos);
 //oos.writeObject(marshalledNode);

 //ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
 //ps.setObject(1, bais, baos.size());
 ps.setObject(1, new MarshalledValue(node));
 }

Hi

Yeah, I feel your pain - I remember many a hair-tearing evening trying to debug stuff only to find that same bug you speak of in MySQL's driver.

Thanks for the fix, we'll have a look at how portable it is across DBs and see how we can implement it.

Until then, we may just stick the patch up on a wiki page or something if someone needs a quick fix pertaining to this particular MySQL driver.

Cheers,
Manik

By the way, have a look at http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossCache for details on the tags and branches of various versions.

I must be looking in the wrong place...

Connected to: anoncvs.forge.jboss.com
from within eclipse

I don't see the tag:

BRANCH_JBOSSCACHE_1_2_3_FINAL

Under Branches

under versions JBossCache
or under versions jboss-cache

I must be looking in the wrong place.

Can we wrap this into a JIRA issue (attach the patches), so we don't forget about it ?

http://jira.jboss.com/jira/browse/JBCACHE-320

Thanks, DashV

The latest from CVS does indeed fix the JBossAS service dependancy problems. I am now able to boot our configuration with optimistic locking enabled. So far I have the following things to note:

1) If optimistic locking is enabled on a cacheloader enabled cache access to that cache becomes painfully slow. I mean on the order of several minutes. I think its the initial load itself that is bogging down. But even after that just a write is quite painful requiring minutes to complete.

2) With optimistic locking enabled for my non cacheloader based REPL_SYNC cache using org.jboss.cache.JBossTransactionManagerLookup as my TransactionManagerLooup class I got the following problem to occur after ~100(give or take 30) interactions with cache:

12:25:25,307 INFO [OptimisticValidatorInterceptor] DataNode
name=SequenceStats
fqn=/SystemCore/Connections/4b6s1o1j-zp0y3-eeodjsph-1-eeodtrdm-5/Sequences/4b6s1o1j-zp0y3-eeodjsph-1-eeodtrkk-6/SequenceStats
data={FirstSend=1129047703349, TotalSends=63, LastSend=1129047925291}
 read locked=false
 write locked=true
(Optimistically locked node)
 version number (130) is greater than or equal to workspace node version 130
12:25:25,307 WARN [OptimisticTxInterceptor] runPreparePhase() failed. Transaction is marked as rolled back
org.jboss.cache.CacheException: unable to validate nodes
 at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.validateNodes(OptimisticValidatorInterceptor.java:115)
 at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.invoke(OptimisticValidatorInterceptor.java:70)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41)
 at org.jboss.cache.interceptors.OptimisticLockingInterceptor.invoke(OptimisticLockingInterceptor.java:87)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41)
 at org.jboss.cache.interceptors.OptimisticReplicationInterceptor.invoke(OptimisticReplicationInterceptor.java:76)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:41)
 at org.jboss.cache.interceptors.OptimisticTxInterceptor.handleLocalPrepare(OptimisticTxInterceptor.java:262)
 at org.jboss.cache.interceptors.OptimisticTxInterceptor.access$000(OptimisticTxInterceptor.java:30)
 at org.jboss.cache.interceptors.OptimisticTxInterceptor$SynchronizationHandler.beforeCompletion(OptimisticTxInterceptor.java:606)
 at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:72)
 at org.jboss.tm.TransactionImpl.doBeforeCompletion(TransactionImpl.java:1384)
 at org.jboss.tm.TransactionImpl.beforePrepare(TransactionImpl.java:1076)
 at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:296)
 at org.jboss.tm.TxManager.commit(TxManager.java:200)
 at org.jboss.cache.interceptors.OptimisticTxInterceptor.handleLocalTx(OptimisticTxInterceptor.java:216)
 at org.jboss.cache.interceptors.OptimisticTxInterceptor.invoke(OptimisticTxInterceptor.java:109)
 at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:4305)
 at org.jboss.cache.TreeCache.put(TreeCache.java:3022)
 at org.jboss.cache.TreeCache.put(TreeCache.java:2963)
 at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
 at mil.af.rl.jbi.platform.server.util.services.cache.impl.JBossSerializableCacheServiceHelper.invokeServerMethod(JBossSerializableCacheServiceHelper.java:266)
 at mil.af.rl.jbi.platform.server.util.services.cache.impl.JBossSerializableCacheServiceHelper.put(JBossSerializableCacheServiceHelper.java:183)
 at mil.af.rl.jbi.platform.server.util.cache.impl.SerializableJBossCacheMap.put(SerializableJBossCacheMap.java:374)
 at mil.af.rl.jbi.platform.server.util.cache.CachedMap.put(CachedMap.java:255)
 at mil.af.rl.jbi.platform.shared.capi.core.plugin.j2ee.jms.impl.SubProliferator.disseminate(SubProliferator.java:223)
 at mil.af.rl.jbi.platform.server.publication.core.impl.PluggableDisseminator.disseminate(PluggableDisseminator.java:84)
 at mil.af.rl.jbi.platform.server.publication.core.utils.impl.PSDisseminationWorker.exec(PSDisseminationWorker.java:270)
 at mil.af.rl.jbi.platform.server.publication.core.utils.impl.PSDisseminationWorker.run(PSDisseminationWorker.java:350)
 at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)

I also have an issue with the 1.2.4 beta. I get an AopOperationNotSupportedException when I try to obtain a persisted TreeCacheAop , like in this (partial) stacktrace:

caused by: org.jboss.cache.aop.AopOperationNotSupportedException: CachedMapImpl: map.values() operation not supported
 at org.jboss.cache.aop.collection.CachedMapImpl.values(CachedMapImpl.java:171)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at org.jboss.cache.aop.collection.CollectionInterceptorUtil.invoke(CollectionInterceptorUtil.java:122)
 at org.jboss.cache.aop.collection.CachedMapInterceptor.invoke(CachedMapInterceptor.java:108)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:88)
 at AOPClassProxy$0.values(AOPClassProxy$0.java)
 at nl.mindef.c2sc.titaan.tbl.server.TblServerRmt.getFeeds(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
 at sun.rmi.transport.Transport$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)

BTW: is it possible to get a POJO that has just been created? I try to use the TreeCacheListener like tthis:

public void nodeCreated(Fqn fqn) {
 Object pojo = cache.getObject(fqn);
}

Motormind,

You've been so helpful in the beta, we wanted to find a way to get Map.values() working for you in 1.2.4. So, we pushed that one forward from 1.3 and its fixed in CVS. See

http://jira.jboss.com/jira/browse/JBCACHE-342

Thanks much for all your helpful posts.

DashV,

If you have some time, could you please try your optimistic locking tests again with 1.2.4 release? O/L it is still beta quality even in release 1.2.4, but feedback would be great to help solidify it for 1.3.

The performance problem with cache loaders should have improved,and the behaviour you see (validation exceptions) after several iterations should be solved as well.

Cheers,
Manik