Manik,

Thanks for your reply.

This is not just a distributed lock.  It is basically a backing-store to JMS.  We accept JMS messages from various sources and need to reliably deliver them to our clients, again through JMS.  Only one system should pub any particular message to the clients.

Placing the messages into the cache is done by locking the key and then doing the putIfAbsent within a transaction.  A CacheListener awakens seeing a CacheEntryCreated event.  The listener then locks on the key and, if the cache still contains the key (may have been deleted by the other system) pubs the message and deletes the cache entry, all within a transaction.  This should allow only one system to pub the message to the clients.

As in the original post, the cluster-wide locking we were expecting does not seem to happen.

We experimented with SEQUENCE and also CENTRAL/PEER_LOCKing.  Thanks for the pointers.  However, we are getting deadlock exceptions.  Config details below.

Then we experimented with setting EagerLockSingleNode to true and we seem to get the desired behavior – surprisingly.  We are working in a two-system cluster and EagerLockSingleNode says it performs a lock to one remote system.  It seems to have given us the cluster-wide (two system) locking behavior we need.  Are we fooling ourselves?

We are configuring the cache with a file patterned off of the example config files with our additions and minor mods:

<?xml version="1.0" encoding="UTF-8"?>

<infinispan

      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

      xsi:schemaLocation="urn:infinispan:config:4.2 http://www.infinispan.org/schemas/infinispan-config-4.2.xsd"

      xmlns="urn:infinispan:config:4.2">

   <global>

      <globalJmxStatistics

            enabled="true"

            jmxDomain="org.infinispan"

            cacheManagerName="SampleCacheManager"/>

      <transport

            clusterName="infinispan-cluster"

            machineId="m1-DJR"

            rackId="r1" nodeName="Node-A-DJR">

         <properties>

            <property name="configurationFile" value="jgroups-tcp.xml" />

         </properties>

      </transport>

   </global>

   <default>

      <locking

         isolationLevel="READ_COMMITTED"

         lockAcquisitionTimeout="20000"

         writeSkewCheck="false"

         concurrencyLevel="5000"

         useLockStriping="false"

      />

      <transaction

            transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"

            syncRollbackPhase="true"

            syncCommitPhase="true"

            useEagerLocking="true"

            eagerLockSingleNode="true"

            cacheStopTimeout="30000" />

      <deadlockDetection enabled="true" spinDuration="1000"/>

      <jmxStatistics enabled="true"/>

      <clustering mode="replication">

         <stateRetrieval

            timeout="20000"

            fetchInMemoryState="false"

            alwaysProvideInMemoryState="false"

         />

         <sync replTimeout="20000"/>

      </clustering>

   </default>

    <namedCache name="OUR_CACHE">

      <clustering mode="distribution">

         <sync/>

         <hash

            numOwners="2"

            rehashWait="120000"

            rehashRpcTimeout="600000"

         />

         <l1

            enabled="false"

            lifespan="600000"

         />

      </clustering>

   </namedCache>

</infinispan>

The jgroups-tcp.xml is taken from the examples as well.  It is in there that we added the SEQUENCE block:

   <SEQUENCER ergonomics="false"

              level="TRACE"

   />

We tried this block at both the top and bottom of the config block, but continued to have deadlock issues.  We saw the jgroups sequencer in the logged configuration for jgroups, but nowhere else.

How can we best achieve our goals?

Manik,

Thanks for the pointer and warning about putIfAbsent.  We read many of the posts and had high hopes that replacing putIfAbsent() with a simpler put() would suddenly make everything work.  Unfortunately, we continued to have problems - some new and soluble (e.g. ClassNotFound on UUID) and others related to locking.

We're back to wondering if we are trying to make Infinispan do things for which it was not designed.  It seems that having a few writers to the cache, many readers and fairly long data lifetimes is a supported paradigm.

We have two writers, two readers and data lifetimes in the milliseconds.  We anticipated that eager locking would slow us down, but we expected it to work.  We can't get it to work.

Are we trying to put a square peg in a round hole?

Thanks again.