Distributed and replicated cache are provided as local

[sgarcia]

Hi everyone,

I'm trying to prepare a clustered environment using jboss as7.1.1.Final running in domain mode.

I have a distributable war that uses a cache defined in a cache container. I tried to configure the cache as replicated-cache or distributed-cache, but when I use the cache, each server see it as local cache. I verify not to include infinispan.jar in the war and I added the dependency reference in MANIFEST.MF to use the module bundled of the jboss server.

My Cache subsystem configuration is:

<cache-container name="cluster" aliases="ha-partition" default-cache="default" eviction-executor="infinispan-eviction" listener-executor="infinispan-listener" replication-queue-executor="infinispan-repl-queue">

                    <transport lock-timeout="60000"/>

                    <replicated-cache name="default" mode="SYNC" start="EAGER" batching="true">

                        <locking isolation="REPEATABLE_READ"/>

                    </replicated-cache>

                    <distributed-cache name="control" owners="0" mode="SYNC" start="EAGER" batching="true">

                        <locking isolation="READ_COMMITTED"/>

                        <file-store/>

                    </distributed-cache>

</cache-container>

I receive the following debuggin information:

17:54:16,398 INFO  [stdout] (MSC service thread 1-16) Configuration{classLoader=ModuleClassLoader for Module "deployment.xtent.war:main" from Service Module Loader, clustering=ClusteringConfiguration{async=AsyncConfiguration{asyncMarshalling=false, replicationQueue=org.infinispan.remoting.ReplicationQueueImpl@34f2c18e, replicationQueueInterval=5000, replicationQueueMaxElements=1000, useReplicationQueue=false}, cacheMode=LOCAL, hash=HashConfiguration{activated=false, consistentHash=null, hash=org.infinispan.commons.hash.MurmurHash3@2d816cca, numOwners=2, numVirtualNodes=48, groupsConfiguration=GroupsConfiguration{enabled=false, groupers=[]}, stateTransferConfiguration=StateTransferConfiguration{chunkSize=10000, fetchInMemoryState=false, originalFetchInMemoryState=false, timeout=240000}}, l1=L1Configuration{activated=true, enabled=false, invalidationThreshold=0, lifespan=600000, onRehash=false}, stateTransfer=StateTransferConfiguration{chunkSize=10000, fetchInMemoryState=false, originalFetchInMemoryState=false, timeout=240000}, sync=SyncConfiguration{replTimeout=15000}}, customInterceptors=CustomInterceptorsConfiguration{interceptors=[]}, dataContainer=DataContainerConfiguration{dataContainer=null}, deadlockDetection=DeadlockDetectionConfiguration{enabled=false, spinDuration=100}, eviction=EvictionConfiguration{maxEntries=-1, strategy=NONE, threadPolicy=DEFAULT}, expiration=ExpirationConfiguration{lifespan=-1, maxIdle=-1, reaperEnabled=true, wakeUpInterval=60000}, indexing=IndexingConfiguration{enabled=false, indexLocalOnly=false}, invocationBatching=InvocationBatchingConfiguration{enabled=false}, jmxStatistics=JMXStatisticsConfiguration{enabled=false}, loaders=LoadersConfiguration{cacheLoaders=[], passivation=false, preload=false, shared=false}, locking=LockingConfiguration{concurrencyLevel=32, isolationLevel=READ_COMMITTED, lockAcquisitionTimeout=10000, useLockStriping=false, writeSkewCheck=false}, storeAsBinary=StoreAsBinaryConfiguration{enabled=false, storeKeysAsBinary=true, storeValuesAsBinary=true}, transaction=TransactionConfiguration{autoCommit=true, cacheStopTimeout=30000, eagerLockingSingleNode=false, lockingMode=OPTIMISTIC, syncCommitPhase=true, syncRollbackPhase=false, transactionManagerLookup=org.infinispan.transaction.lookup.GenericTransactionManagerLookup@c0d591d, transactionSynchronizationRegistryLookup=null, transactionMode=NON_TRANSACTIONAL, useEagerLocking=false, useSynchronization=false, recovery=RecoveryConfiguration{enabled=false, recoveryInfoCacheName='__recoveryInfoCacheName__'}, use1PcForAutoCommitTransactions=false}, versioning=VersioningConfiguration{enabled=false, scheme=NONE}, unsafe=UnsafeConfiguration{unreliableReturnValues=false}}

17:54:16,416 INFO  [stdout] (MSC service thread 1-16) Cluster name ISPN

17:54:16,417 INFO  [stdout] (MSC service thread 1-16) Cluster members local

I highlighted the cacheMode=LOCAL.

I'd really appreciate any kind of support.

Regards.

[pferraro]

Are you sure that your server group uses "ha" or "full-ha" profile?  I ask because the non-ha profiles use a local cache by default for <distributable/> applications.

Also - web session clustering does not use the "cluster" cache container, but rather the "web" cache container.

[sgarcia]

I'm using full-ha profile.

    <server-groups>

        <server-group name="mycluster" profile="full-ha">

            <jvm name="default">

                <heap size="64m" max-size="512m"/>

            </jvm>

            <socket-binding-group ref="full-ha-sockets"/>

            <deployments>

            </deployments>

        </server-group>

    </server-groups>

I access cluster cache container through jndi in my code. I supose it's possible to define more cache containers independently of the predefines one in order to group application caches we are going to use. First of all, I tried to define a application cache container and after seeing it doesn't work I tried to use the cluster one, that is the instance I see when cluster is runing with <distributable/> tag in .war.

Configuration files are attached to https://community.jboss.org/thread/220985

I read an article that refers to https://github.com/jbossas/jboss-as/blob/master/clustering/infinispan/src/main/resources/infinispan-defaults.xml as default settings of infinispan caches defined in infinispan subsystem. It appears cacheModel as LOCAL and transactionMode=NON_TRANSACTIONAL. Maybe thats the reason, if thats the origin of this issue, is there any way to change these properties in the submodule.

Regards.

[pferraro]

What do you mean by "I tried to define a application cache container and after seeing it doesn't work"?  How did you define an application cache container?  How did you determine that "it doesn't work"?

The defaults in "infinispan-defaults.xml" have *nothing* to do with the default cache mode of a given cache container.  In AS7 the "default" cache of a cache container is explicitly defined by the "default-cache" attribute.  Consequently, the cache returned by CacheContainer.getCache() is a cache instance based on the configuration defined by the designated "default-cache" of the cache-container.

Perhaps it would be best if you could paste the code that access the cache container via jndi?  To best diagnose your issue, I think we need to start there.

[sgarcia]

I created a specific cache container to use it in my application, not depending on default ones defined in domain.xml. When I said it doesn't work I meant that the cache is also identified as cacheMode=LOCAL.

I'm using spring 3.2 and I have the following configuration in application context:

	<jee:jndi-lookup id="cacheContainer" jndi-name="java:jboss/infinispan/container/xtent"/>
	<jee:jndi-lookup id="txManager" jndi-name="java:/jboss/TransactionManager"/>

	<infinispan:container-cache-manager cache-container-ref="cacheContainer"/>
	<bean id="cacheContainer" class="org.infinispan.manager.DefaultCacheManager"/>

<jee:jndi-lookup id="connectionfactory" jndi-name="java:/JmsXA" resource-ref="false"/>

    <jee:jndi-lookup id="destination" jndi-name="java:/queue/test" resource-ref="false"/>

	<int:channel id="jmsChannel"/>
	<bean id="applicationLifeCycleManager" class="es.seinet.xtent.messaging.producers.ApplicationLifeCycleManager"/>
	<int:service-activator input-channel="jmsChannel" ref="applicationLifeCycleManager" method="upperCase"/>

	<task:scheduler id="heartbitScheduler" pool-size="1"/>
	<task:scheduled-tasks scheduler="heartbitScheduler">
	<task:scheduled ref="applicationLifeCycleManager" method="sendMessage" fixed-rate="1000"/>

    </task:scheduled-tasks>

Class ApplicationLifeCycleManager

package es.seinet.xtent.messaging.producers;

import java.util.Random;

import javax.inject.Inject;

import org.infinispan.Cache;

import org.infinispan.manager.DefaultCacheManager;

import org.springframework.integration.MessageChannel;

import org.springframework.integration.annotation.MessageEndpoint;

import org.springframework.integration.annotation.ServiceActivator;

import org.springframework.integration.support.MessageBuilder;

import org.springframework.stereotype.Component;

@Component

@MessageEndpoint

public class ApplicationLifeCycleManager {

          @Inject

          private MessageChannel jmsChannel;

          @Inject

          private DefaultCacheManager cacheContainer;

          public void sendMessage() {

                    Cache controlCache = cacheContainer.getCache("control");

                    Integer cont = (Integer) controlCache.get("msg.cont");

                    if (cont == null) {

                              cont = Integer.valueOf(0);

                    }

                    cont++;

                    controlCache.put("msg.cont", cont);

              jmsChannel.send(MessageBuilder.withPayload("Mensaje " + cont ).build());

        System.out.println("Heartbit Sent - " + "Mensaje " + cont);

        try {

                              wait(new Random().nextInt(200));

                    } catch (InterruptedException e) {

                              // TODO Auto-generated catch block

                              e.printStackTrace();

                    }

    }

          @ServiceActivator(inputChannel="jmsChannel")

          public void upperCase(String input) {

                    System.out.println("JMS response: " + input.toUpperCase());

          }

}

Domain.xml

                <cache-container name="xtent">

                    <transport stack="udp" lock-timeout="60000"/>

                    <replicated-cache name="control" mode="SYNC" start="EAGER" batching="true">

                        <transaction mode="FULL_XA" locking="PESSIMISTIC"/>

                    </replicated-cache>

                </cache-container>

When I ran the application in more than one server instance, the cache is recognized as local mode.

[pferraro]

Do you have infinispan jars in your application archive?  If so, the cache manager you'll get is not the one from the infinispan subsystem, but rather some default from Infinispan itself.  The cache managers returned by the infinispan subsystem are *not* instances of org.infinispan.manager.DefaultCacheManager.

You don't need spring to inject the cache containers from AS7 into your application.

The following will do exactly what you want:

@Resource(lookup = "java:jboss/infinispan/container/xtent")

private EmbeddedCacheManager cacheContainer;

Using this approach, you should also not use start="EAGER".  That way your cache container will start automatically when your application is deployed, and stop when it is undeployed.

[sgarcia]

Thanks for your support. The issue was related to DefaultCacheManager. I changed it for EmbeddedCacheManager and it works.

Do you know if I need any thing more to use transactions and locking cache keys?

Regards.

[pferraro]

By default, caches are transactional and can be locked pessimistically using the AdvancedCache api (i.e. cache.getAdvancedCache().lock(...)).