2 Replies Latest reply on Jul 16, 2014 11:45 AM by Grégory Bataille

    Cluster: locate giving inconsistent values

    Grégory Bataille Newbie

      Hi guys,

       

      so I have a cluster of 4 wildfly nodes (running Infinispan 6.0.1).

      I have a distributed cache configured with 2 owners

                          <distributed-cache name="rtpm-cache-timeserie" start="EAGER" batching="true" mode="ASYNC" owners="2"/>

       

      I have 3 objects I'm playing with in my store which have keys "YHOO", "AAPL" and "GOOG"

      I am trying to use interceptors and to do something specific in those when I'm on a node that owns the data being touched.

              String owner = timeSerieCache.getAdvancedCache().getDistributionManager().locate(command.getKey()).toString();

              String server = System.getProperty("jboss.server.name");

              if (owner.contains(server)) {...}

      Everything works fine in a standalone setup or in a 2 nodes distributed setup with a single cache owner.

       

      Now in my test with 4 nodes, distributed cache, 2 owners:

      • the first 2 keys behave as expected in a consistent fashion. Whatever server I use as an entry point, I can play with my values, and with some jconsole lookup, the values are on 2 nodes as expected.
      • now for the "GOOG" key, I experience 2 issues (that are most certainly linked)
        • First, when I write the value to this key for the first time, it is actually written in a single node (node 2 in my case) according to the JMX jconsole lookup
        • More importantly, and I think that's the root of the evil, the locate call behaves completely differently depending where I initiate it from
          • on node 1, locate gives me Owner [master:server-three/rtpm-cache-container, master:server-four/rtpm-cache-container]
          • on node 2, locate gives me Owner [master:server-three/rtpm-cache-container, master:server-two/rtpm-cache-container]
          • on node 4, locate gives me Owner [master:server-three/rtpm-cache-container, master:server-four/rtpm-cache-container]
          • on node 3, locate gives me Owner [master:server-one/rtpm-cache-container, master:server-two/rtpm-cache-container]

       

      Obviously when I try then to do a get, the cache calls 2 nodes for help (whichever owner it thinks are the good ones) and it returns null because it either interrogates the 2 wrong nodes, or it interrogates the right one and a false one, and the false one having no value, it answers first!!!

       

      Any clue what I can do? is there a way to easily debug the Hashing class (I'm using the default distribution algorithm, haven't touched anything there)

       

      Another information, the keys I am using are not string but part of an enum actually. In case that changes anything...

       

      public enum Ticker implements Serializable {

          IBM("IBM"),

          MSFT("MSFT"),

          GOOG("GOOG"),

          AAPL("AAPL"),

          YHOO("YHOO"),

          UBS("UBS.N"),

          CS("CS.N"),

          DB("DB"),

          NESN("NESN.VX"),

          PG("PG.N"),

          SNE("SNE");

       

       

          private static final long serialVersionUID = 1l;

       

       

          private Ticker(final String ticker) {

              this.ticker = ticker;

          }

          private final String ticker;

       

       

          @Override

          public String toString() {

              return ticker;

          }

       

       

          public static Ticker fromString(String text) throws IllegalArgumentException {

              if (text != null) {

                  for (Ticker b : Ticker.values()) {

                      if (text.equalsIgnoreCase(b.ticker)) {

                          return b;

                      }

                  }

              }

              // if it reaches here, that means the ticker string used is invalid

              throw new IllegalArgumentException("Ticker String invalid");

          }

       

       

      }