3 Replies Latest reply on Feb 29, 2012 3:24 AM by galder.zamarreno

    How make the distributed storage in sync after an outage?

    jbossas7

      I have 3 Infinispan servers set up in distribution mode with 2 owners and each has its own cache store like a file on a disk. The cache loader is configured to allow pre-load.

      (1) Application on server #1 inserted a key/value pairs, (k1, v1), into the cache. Looking at the files on the 3 servers, both files on server #1 and #2 contains (k1, v1) as expected.

      (2) Stopped server #2 to simulate a hardware failure.

      (3) Application on server #1 removed k1 from the cache. The cache was empty. Checking the files again and found only the file on server #2 still has(k1, v1).

      (4) Started server #2 again to simulate that the hardware problem had been solved. However, it brought (k1, v1) back.

       

      How make the distributed storage in sync after an outage?

        • 1. Re: How make the distributed storage in sync after an outage?
          galder.zamarreno

          Hmmm, you could enable fetchPersistentState in the cache loader configuration so that when a node starts up, it overrides it's persistent state with the in-memory state of the cluster. That only works in your scenario where each node has its own cache store. See http://docs.jboss.org/infinispan/5.1/configdocs/urn_infinispan_config_5.1/complexType/configuration.loaders.loader.html

           

          Alternatively, use a share cache store and you won't have that problem, where all nodes point to the same database for example.

          • 2. Re: How make the distributed storage in sync after an outage?
            jbossas7

            Galder, thanks for your respose. However the result is not what I'm expecting even wiht fetchPersistentState enabled.

             

            Here is my configuration:

             

            <?xml version="1.0" encoding="UTF-8"?>

            <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

                xmlns="urn:infinispan:config:5.1">

             

                <global>

                    <transport

                        transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport">

                        <properties>

                            <property name="configurationFile" value="jgroups-tcp.xml" />

                        </properties>

                    </transport>

                </global>

             

                <default>

                    <!-- Configure a synchronous replication cache -->

                    <clustering mode="distribution">

                        <sync />

                        <hash numOwners="2" />

                    </clustering>

                </default>

             

                <namedCache name="Demo">

                    <loaders passivation="false" shared="false" preload="true">

                        <loader class="org.infinispan.loaders.file.FileCacheStore"

                            fetchPersistentState="true" ignoreModifications="false"

                            purgeOnStartup="false">

                            <properties>

                                <property name="location" value="/tmp/ispn/store" />

                            </properties>

                        </loader>

                    </loaders>

                    <eviction strategy="LIRS" maxEntries="5" />

                </namedCache>

             

            </infinispan>

             

            My simple application takes inputs from the console and does INSERT (to the cache), REMOVE (from the cache) and GET (the value from the cache). I run it on 3 different machines. Here is what I had

            1. On server #1 I let the application to insert (k1, v1)  into the cache and saw the pair was distributed to the files on both server #1 and #2. I could get v1 with Cache.get("k1") from ANY of the 3 consoles.
            2. Then I stopped the instance on server #2 with (k1, v1) left in its file.
            3. Again on server #1 I let the application to remove (k1, v1) from the cache, which also removed the pair from the file on server #1.
            4. So far so good.
            5. On server #2, I restarted the application.
            6. On each server, I let the application to get the value with Cache.get("k1"). Both server #1 and #3 returned nothing as expected by server #2 returned v1 back again.

             

            My questions are:

            1. If server #2 reloaded (k1, v1) from its file to the cache, why did I fail to get it from server #1 and #3?
            2. I'm expecting server #2 can sync up with the cache on startup, but it seems it doesn't even with fetchPersistentState enabled.

             

            Did I miss anything else here? Thanks in advance.

            • 3. Re: How make the distributed storage in sync after an outage?
              galder.zamarreno

              Re 1. Had you already retrieved data from #2 when you requested data from #1 and #3? If it was the other way around, it makes sense for #1 to not look in all other nodes cache stores cos this would be expensive.

               

              Re 2. Hmmm, I think fetchPersistentState does not work with distribution mode. It only works with replicated or invalidation modes.

               

              It's probably best if you shared a cache store if you're using distribution as clustering mode