Help debugging NearCache and multiple HotRod server architecture

[enrico.olivelli]

Hi,

I'm running Infinispan 7.2.3Final and I'm in trouble with the nearcache.

That fact is that when running a cluster with more than one server after some time (I'm not yet able to reproduce the problem systematically) all the Java HotRod Clients clients stop behaving correctly, that is, it reads "stale" values from Infinispan.

This is the case, in the simplest staging environment which reproduces the problem:

- two java hotrod clients

- two java hotrod servers (embedded in a java application, which does nothing but hosting the Infinispan server)

- the clients are configured to connected to all the servers

the client follows these steps:

- writes an entry to a cache, with value=A

- updates the entry to the cache, with value=B

- reads the entry from the cache, the returned value is A

of course this does not happens always, most of the time Infinispan works as expected, but after some time that the system is fully running every client stops behaving as expected and the buggy beheaviur  starts.

Even after restarting the server nodes the clients are not able to recover anymore, the only resolution is to restart the clients.

Even putting  all the loggers at DEBUG level I cannot find any "exception", both on servers and on clients.

Any idea of how can I debug this situation ?

client side bootstrap code:

            org.infinispan.client.hotrod.configuration.ConfigurationBuilder clientBuilder = new org.infinispan.client.hotrod.configuration.ConfigurationBuilder().forceReturnValues(false);

                clientBuilder.nearCache().mode(NearCacheMode.LAZY).maxEntries(100000);

            clientBuilder

                    .connectionPool()

                    .exhaustedAction(ExhaustedAction.WAIT)

                    .maxActive(10)

                    .maxIdle(10)

                    .maxTotal(20)

                    .minIdle(1)

                    .minEvictableIdleTime(60000)

                    .timeBetweenEvictionRuns(60000)

                    .numTestsPerEvictionRun(20)

                    .testOnBorrow(false)

                    .testOnReturn(false

                    .testWhileIdle(true);

            clientBuilder

                    .socketTimeout(sockettimeout)

                    .connectionTimeout(connecttimeout);

                clientBuilder.

                        security()

                        .authentication()

                        .enable()

                        .serverName("localhost")

                        .saslMechanism("CRAM-MD5")

                        .callbackHandler(new SecurityCallbackHandler(username, password, realm));

                 for each server...{

                    clientBuilder

                            .addServer()

                            .host(serveraddress)

                            .port(serverport);

                }

RemoteCacheManager remoteCacheManager = new RemoteCacheManager(clientBuilder.build());

Server side configuration:

GlobalConfiguration globalConfig = new GlobalConfigurationBuilder()

                    .globalJmxStatistics()

                    .allowDuplicateDomains(true)

                    .cacheManagerName(instanceName)

                    .transport()

                    .defaultTransport()

                    .clusterName(clustername)

                    .addProperty("configurationFile", configurationFile)   // using default jgroups UDP multicast

                    .machineId(instanceName)

                    .siteId("xxxx")

                    .rackId("xxxxx")

                    .nodeName("uniquenamexxxx")

                    .build();

            Configuration wildcard = new ConfigurationBuilder()

                    .locking().lockAcquisitionTimeout(lockAcquisitionTimeout)

                    .concurrencyLevel(10000).isolationLevel(IsolationLevel.READ_COMMITTED).useLockStriping(true)

                    .clustering()

                    .cacheMode(CacheMode.DIST_SYNC)

                    .l1().lifespan(l1ttl).enabled(l1ttl > 0)                            // problems happens both with l1 that without l1

                    .hash().numOwners(1).capacityFactor(1)

                    .partitionHandling().enabled(false)

                    .stateTransfer().awaitInitialTransfer(false).timeout(initialTransferTimeout).fetchInMemoryState(false)

                    .storeAsBinary().enabled(false).storeKeysAsBinary(false).storeValuesAsBinary(false)

                    .jmxStatistics().enable()

                    .unsafe().unreliableReturnValues(false)

                    .compatibility().enable()

                    .build();

EmbeddedCacheManager manager = new DefaultCacheManager(globalConfig, wildcard, false);

HotRodServer hotrodserver =new HotRodServer();

HotRodServerConfigurationBuilder configBuilder = new HotRodServerConfigurationBuilder()

                        .host(address)

                        .port(port);

SimpleServerAuthenticationProvider authProv = new SimpleServerAuthenticationProvider();

authProv.addUser(username, realm, password);

configBuilder.authentication().addAllowedMech("CRAM-MD5").serverAuthenticationProvider(authProv).serverName("localhost").enable();

hotrodserver.start(configBuilder.build(),manager);

[rvansa]

The near cache functionality is based on remote listeners, therefore it is updated from the async notifications in the listeners.

I would use jstack first to see if there's any suspicious thread trying to push something to a full queue, write to a socket etc. DEBUG level is usually not enough to debug (although the name could suggest...), TRACE level provides the finest messages. I know there are a lot of them. And regrettably it's not about exceptions, you need to go through the traces to see where the invocation of the write ended, not triggering the listeners.

[enrico.olivelli]

Thank you, I will follow your suggestion as soon as the problems comes up again.

(My previous post was wrong, I enabled ALL Level in order to see all the logs, not the DEBUG level)

[enrico.olivelli]

One question, surely I'm wrong, but I missed something of how the nearcache works

the  NearRemoteCache reads values from the "nearcache", but when I write/remove entries they are not updated/removed from the nearcache ? If it is so, the client can write some data and then read previous values if the listener does not receives updates in time ?

Would it be better to at least "invalidate" entries in the nearcache as long as the client updates/remove an entry ?

[rvansa]

I am no expert at near cache, but quick look into source code shows that you may be right. Such behaviour would make more sense to me (the value in near cache should be at least invalidated upon write - double invalidation can't hurt). Could you file a JIRA with this request?

[enrico.olivelli]

Here it is:

https://issues.jboss.org/browse/ISPN-5607

Thank you