Locks not working as expected in DIST_SYNC cache
daver32 Jul 5, 2011 6:49 PMWell, we made it past the joining, but we seem to be missing something on locking. It seems that we cannot achieve cluster-wide locking.
We are running Infinispan within a JBoss AS 5.1.0.GA cluster of two systems. Our application accepts JMS messages and places them in the Infinispan cache. Listeners to the cache on each system awaken and try to lock the just-added node, process it and delete it.
The logic in the listener is as follows:
public void newEntry( CacheEntryCreatedEvent cece ) {
TransactionManager tm = cache.getAdvancedCache().getTransactionManager();
tm.begin();
String key = cece.getKey().toString();
cache.getAdvancedCache().lock(key); // Hoping that this is a cluster-wide lock - one node gets it, the other waits
if (cache.containsKey(key)) { // May have been removed by listener on the other node
... publish on particular JMS topic ...
cache.remove(key);
}
tm.commit(); // Releases lock
}
Unfortunately, we are regularly getting errors like the following:
2011-07-05 15:33:28,129 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (OOB-1,MY_CLUSTER,MyHost-15542) Execution error:
org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [ID:JBM-5d328a73-3b57-4a43-b3f4-acc677b1e1e3] for requestor [GlobalTransaction:<OtherHost-43220>:9:remote]! Lock held by [GlobalTransaction:<MyHost-15542>:5:local]
at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:228)
at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:155)
...
This confuses us. The work we do while the key is locked is sub-second work.
We often see that both nodes get the error on the same key.
The cache is configured as follows:
GlobalConfiguration gc = GlobalConfiguration.getClusteredDefault();
gc.setClusterName( "MY_CLUSTER" );
Properties p = new Properties();
p.setProperty("configurationFile", "jgroups-tcp.xml"); // Not sure this is required
gc.setTransportProperties(p);
Configuration c = new Configuration();
c.setCacheMode( Configuration.CacheMode.DIST_SYNC ); // ********
c.setUseEagerLocking(true); // Seems necessary
c.setSyncCommitPhase(true);
c.setSyncRollbackPhase(true);
// c.setUseReplQueue(false);
c.setIsolationLevel(IsolationLevel.READ_COMMITTED);
c.setTransactionManagerLookupClass("org.infinispan.transaction.lookup.GenericTransactionManagerLookup" );
MarshallerFactory foo = (MarshallerFactory) Thread.currentThread().getContextClassLoader()
.loadClass("org.jboss.marshalling.river.RiverMarshallerFactory").newInstance();
EmbeddedCacheManager cm = new DefaultCacheManager( gc, c );
cache = cm.getCache( "MY_CACHE" );
What have we missed? It seems that the cluster-wide locking is not working for us.
Is our model not a good one for Infinispan use? Is there something we could do to better match Infinipsan's strengths?
Are we just not configuring things correctly to achieve cluster-wide locking?
Are we misuing the transactions or lock calls?
Any hints will be appreciated.