Cluster: locate giving inconsistent values
Hi guys,
so I have a cluster of 4 wildfly nodes (running Infinispan 6.0.1).
I have a distributed cache configured with 2 owners
<distributed-cache name="rtpm-cache-timeserie" start="EAGER" batching="true" mode="ASYNC" owners="2"/>
I have 3 objects I'm playing with in my store which have keys "YHOO", "AAPL" and "GOOG"
I am trying to use interceptors and to do something specific in those when I'm on a node that owns the data being touched.
String owner = timeSerieCache.getAdvancedCache().getDistributionManager().locate(command.getKey()).toString();
String server = System.getProperty("jboss.server.name");
if (owner.contains(server)) {...}
Everything works fine in a standalone setup or in a 2 nodes distributed setup with a single cache owner.
Now in my test with 4 nodes, distributed cache, 2 owners:
- the first 2 keys behave as expected in a consistent fashion. Whatever server I use as an entry point, I can play with my values, and with some jconsole lookup, the values are on 2 nodes as expected.
- now for the "GOOG" key, I experience 2 issues (that are most certainly linked)
- First, when I write the value to this key for the first time, it is actually written in a single node (node 2 in my case) according to the JMX jconsole lookup
- More importantly, and I think that's the root of the evil, the locate call behaves completely differently depending where I initiate it from
- on node 1, locate gives me Owner [master:server-three/rtpm-cache-container, master:server-four/rtpm-cache-container]
- on node 2, locate gives me Owner [master:server-three/rtpm-cache-container, master:server-two/rtpm-cache-container]
- on node 4, locate gives me Owner [master:server-three/rtpm-cache-container, master:server-four/rtpm-cache-container]
- on node 3, locate gives me Owner [master:server-one/rtpm-cache-container, master:server-two/rtpm-cache-container]
Obviously when I try then to do a get, the cache calls 2 nodes for help (whichever owner it thinks are the good ones) and it returns null because it either interrogates the 2 wrong nodes, or it interrogates the right one and a false one, and the false one having no value, it answers first!!!
Any clue what I can do? is there a way to easily debug the Hashing class (I'm using the default distribution algorithm, haven't touched anything there)
Another information, the keys I am using are not string but part of an enum actually. In case that changes anything...
public enum Ticker implements Serializable {
IBM("IBM"),
MSFT("MSFT"),
GOOG("GOOG"),
AAPL("AAPL"),
YHOO("YHOO"),
UBS("UBS.N"),
CS("CS.N"),
DB("DB"),
NESN("NESN.VX"),
PG("PG.N"),
SNE("SNE");
private static final long serialVersionUID = 1l;
private Ticker(final String ticker) {
this.ticker = ticker;
}
private final String ticker;
@Override
public String toString() {
return ticker;
}
public static Ticker fromString(String text) throws IllegalArgumentException {
if (text != null) {
for (Ticker b : Ticker.values()) {
if (text.equalsIgnoreCase(b.ticker)) {
return b;
}
}
}
// if it reaches here, that means the ticker string used is invalid
throw new IllegalArgumentException("Ticker String invalid");
}
}
