Problem getting Infinispan's replicate or distribute modes to work through connector?

[jdbrown]

Environment: Using ModeShape 1.2.0.FINAL and Infinispan 4.0.0.FINAL.  JBossAS6.M2 and deploying to a 2 node cluster on the same physical machine.  Each JBoss node bound to a different IP.  ModeShape app is deployed as a WAR within the /all profile of each JBoss instance.

With a repo backed by an Infinispan cache that is set to replicate, if I update a property to a JCR Node in JBossA, I would expect to see that reflected in JBossB.    I have tried replicated and also distributed modes and neither works.

With JBossA started, when I start JBossB and the repo is initialized, I do see this in the log on JBossA which ensures Infinispan is at least clustering.

{code}INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] AbstractLogImpl (Incoming-2,Taos-2-15384) Received new cluster view: [Taos-2-15384|1] [Taos-2-15384, Taos-2-1280] {code}

However, through the my code that is reading and incrementing the same JCR node/property every 90 seconds, I can see the JCR is not in synch.

The infinispan-config.xml is below:

{code}

<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns="urn:infinispan:config:4.0">
    <global>
        <globalJmxStatistics enabled="true" jmxDomain="infinispan" />
        <transport clusterName="local-infinispan-cluster" />
    </global>
    <default>
        <jmxStatistics enabled="true" />
        <clustering mode="distribution">
            <sync replTimeout="200000" />
            <l1 enabled="true"/>
            <hash numOwners="2" />
        </clustering>
    </default>
</infinispan>

{code}

[jdbrown]

I think I have isololated this as a ModeShape issue versus Infinispan in that in my application, I am first establishing an Infinispan CacheMgr and binding to JNDI, then passing that JNDI name to ModeShape.  So, using that cacheMgr, I can directly set a key/value pair in Infinispan itself using a put("mykey", "myvalue") and have verified that the value is distributed to the other JBoss node.  Same, if the 2nd JBoss node changes that value, it is reflected at the 1st JBoss node as expected.  So, at the base Infinispan level within my application, clustering is working.

So, has anyone had experience wit this use case with ModeShape and trying to cluster the Infinispan backend source?

With ModeShape 1.1.0, I know it was at least trying to copy the data between Infinispan nodes as it was throwing Serialization errors.  Since that was fixed in v1.2.0, those errors are no longer seen, but the data is not being copied between nodes at all.  Any ideas?

[bcarothers]

Hmm.  There's no caching of values on the Infinispan connector, so that's not the issue.  There is caching of values within the JCR session, so that would keep you from seeing updates to existing nodes unless you explicitly refreshed the session.  Somehow, I doubt that's the problem here and that wouldn't have any impact on new nodes (i.e., read non-existent node /a in session 1, create node /a in session 2, read node /a in session 1 should work).

I will take a look at this tonight,  Any chance that you could post your code to give me a bit of a head start?

Thanks for digging into this!

[jdbrown]

Thanks Brian.  I PM'd you my email and I can then email you the source.

[bcarothers]

Got your PM, but I am going to make an educated guess that you did not set the rootNodeUuid property on your sources.  The default rootNodeUuid behavior varies from connector to connector at the moment.  The Infinispan connector happens to use a random UUID.  If you have two sources clustered together with different root node UUIDs, the two sources will share all node data, but path-based lookups won't work.

Disclaimer: I have only tested this at the graph layer so far, but it worked fine consistently once I set the rootNodeUuid to the same value on both sources.  Well, after I set the rootNodeUuid property and after I set -Djava.net.preferIPv4Stack=true.

I am going to open up a defect to add better default behavior for the Infinispan connector rootNodeUuid property and add some documentation to the Reference Guide on how to set this up.

[bcarothers]

I just committed a fix for MODE-764 that changes the default value for the rootNodeUuid to be a constant.  It also adds two test cases (one at the graph level and one at the JCR level) that demonstrate the Infinispan source working in a cluster.  I based it on your configuration from the original post, so you should be good to go after you set the rootNodeUuid property.

Sorry for the delay in figuring this out.  Thanks for catching and reporting this!

[jdbrown]

Thanks Brian, that was it!