5 Replies Latest reply on May 26, 2016 3:39 AM by Juraj Kubica

    Wildfly-10.0.0.CR4 replicated-cache issue.

    Dmitry Kovbenya Newbie

      Hi all,

       

      Seems like replicated-cache does not work for wildfly-10.0.0.CR4, probably missing configuration.

      I was trying to configure infinispan replicated-cache without any success for wildfly-10.0.0.CR4.

       

      Following is my configuration:

       

      1. In the standalone-ha.xml I had defined minimal configuration for app. cache container:

       

      <cache-container name="app_cache" default-cache="userCtx">

          <transport lock-timeout="60000"/>

          <replicated-cache name="userCtx" mode="ASYNC">

              <locking isolation="REPEATABLE_READ"/>

          </replicated-cache>

      </cache-container>

       

       

      2. Application uses Resource injection for interaction with cache:

       

          @Resource(lookup = "java:jboss/infinispan/container/app_cache")

          private EmbeddedCacheManager cacheContainer;

       

      3. MANIFEST.MF has the line: "Dependencies: org.infinispan"

       

      4. WebApp has included <distributable/> tag in web.xml file

       

      5. Then I run 2 nodes just for testing purpose:

      ./bin/standalone.bat -Djboss.server.base.dir=cl-std-tcp-node-1 --server-config=standalone-ha.xml -Djboss.socket.binding.port-offset=100 -Djboss.node.name=node-1

      ./bin/standalone.bat -Djboss.server.base.dir=cl-std-tcp-node-2 --server-config=standalone-ha.xml -Djboss.socket.binding.port-offset=200 -Djboss.node.name=node-2

       

      Then I put some info to the cache for node-1, as result cache data from node-1 do not replicated to node-2.

       

      The same configuration works fine for Wildfly-8.2.0.Final.

       

      Any help will be high appreciated.

        • 1. Re: Wildfly-10.0.0.CR4 replicated-cache issue.
          Paul Ferraro Master

          Can you paste your code that creates your cache from the cacheContainer?

          • 2. Re: Wildfly-10.0.0.CR4 replicated-cache issue.
            Dmitry Kovbenya Newbie

            As I mention above in the standalone-ha.xml config file I define new cache container:

             

            <cache-container name="app_cache" default-cache="userCtx">

                <transport lock-timeout="60000"/>

                <replicated-cache name="userCtx" mode="ASYNC">

                    <locking isolation="REPEATABLE_READ"/>

                </replicated-cache>

            </cache-container>

             

            All interaction with cache performed by ejb singleton:

             

            @Singleton

            public class CacheAccessor

            {

                @Resource(lookup = "java:jboss/infinispan/container/app_cache")

                private EmbeddedCacheManager cacheContainer;

             

               public Cache<String, UserCtx> getUserCtxCache()

                {

                    return cacheContainer.getCache("userCtx");

                }  

            }

            where UserCtx holds some user related data. (implements Serializable)

             

            When I need put\get something to\from the cache I simply inject CacheAccessor

             

            public class AuthManager

            {

                @Inject

                private CacheAccessor cacheAccessor;

                @Inject

                private JwtIssuer jwtIssuer;

               

                public String login(Credentials credentials)

                {

                    UserCtx userCtx = ... initialization logic

                    String token = jwtIssuer.issueToken(credentials.getUsername());

                    // put entry to cache

                    Cache<String, UserCtx> userCtxCache = cacheAccessor.getUserCtxCache();

                    userCtxCache.put(token, userCtx);

                    return token;

                }

             

              

                public void logout(String jwtToken)

                {

                    String subject = jwtIssuer.parseTokenSubject(jwtToken);

                    // remove cache entry

                    Cache<String, UserCtx> userCtxCache = cacheAccessor.getUserCtxCache();

                    userCtxCache.remove(jwtToken);

                }

            }

             

            This works for Wildfly-8.2.0.Final, but not for Wildfly-10.0.0.CR4

            • 3. Re: Wildfly-10.0.0.CR4 replicated-cache issue.
              Paul Ferraro Master

              That's odd - I don't see any problem with the above configuration or code.  Are there any errors in your server log?  If not, try changing the mode to SYNC - does the put succeed?

              • 4. Re: Wildfly-10.0.0.CR4 replicated-cache issue.
                Dmitry Kovbenya Newbie

                There are no errors in server log file.

                I changed mode to SYNC, but the result the same, replication doesn't work between nodes.

                I also tried test with new wildfly-10.0.0.CR5 without any success.

                 

                I enabled DEBUG logging level for "org.infinispan" and "org.jboss.as.clustering.infinispan".

                Then I investigated log files I found record:

                2015-12-24 09:44:39,994 DEBUG [org.infinispan.jmx.JmxUtil] (default task-2) Object name jboss.infinispan:type=Cache,name="userCtx(local)",manager="app_cache",component=Cache already registered

                2015-12-24 09:44:39,994 INFO  [org.jboss.as.clustering.infinispan] (default task-2) WFLYCLINF0002: Started userCtx cache from app_cache container

                2015-12-24 09:44:39,995 DEBUG [org.infinispan.cache.impl.CacheImpl] (default task-2) Started cache userCtx on node-1

                This looks like userCtx cache registered as local. Am I right?

                But log doesn't contain any records about UDP protocol activity.

                 

                I decided make some test with defining my own communication channel.

                I made some changes in standalone-ha.xml:

                1. Define new channel:

                     <subsystem xmlns="urn:jboss:domain:jgroups:4.0">

                            <channels default="ee">

                                <channel name="ee" stack="udp"/>

                               <channel name="my" stack="udp"/>

                            </channels>

                          .......

                        </subsystem>

                2. Change cache container transport:

                     <cache-container name="app_cache" default-cache="userCtx">

                                <transport lock-timeout="60000" channel="my"/>

                                <replicated-cache name="userCtx" mode="ASYNC">

                                    <locking isolation="REPEATABLE_READ"/>

                                </replicated-cache>

                      </cache-container>

                 

                As a result I still have my cache with (local) postfix:

                2015-12-24 10:08:37,674 DEBUG [org.infinispan.jmx.JmxUtil] (default task-1) Object name jboss.infinispan:type=Cache,name="userCtx(local)",manager="app_cache",component=Cache already registered

                 

                But appear some activity by UDP protocol:

                node 1 log:

                2015-12-24 09:53:33,373 WARN  [org.jgroups.protocols.UDP] (INT-1,ee,node-1) JGRP000012: discarded message from different cluster my (our cluster is ee). Sender was node-1 (received 19 identical messages from node-1 in the last 60582 ms)

                2015-12-24 09:53:34,962 WARN  [org.jgroups.protocols.UDP] (INT-2,my,node-1) JGRP000012: discarded message from different cluster ee (our cluster is my). Sender was node-1 (received 17 identical messages from node-1 in the last 62172 ms)

                2015-12-24 09:54:04,971 WARN  [org.jgroups.protocols.UDP] (INT-2,my,node-1) JGRP000012: discarded message from different cluster ee (our cluster is my). Sender was node-2 (received 9 identical messages from node-2 in the last 60076 ms)

                2015-12-24 09:54:04,976 WARN  [org.jgroups.protocols.UDP] (INT-2,ee,node-1) JGRP000012: discarded message from different cluster my (our cluster is ee). Sender was node-2 (received 8 identical messages from node-2 in the last 60083 ms)

                 

                node 2 log

                2015-12-24 09:54:04,962 WARN  [org.jgroups.protocols.UDP] (INT-1,ee,node-2) JGRP000012: discarded message from different cluster my (our cluster is ee). Sender was node-1 (received 17 identical messages from node-1 in the last 60005 ms)

                2015-12-24 09:54:04,962 WARN  [org.jgroups.protocols.UDP] (INT-2,my,node-2) JGRP000012: discarded message from different cluster ee (our cluster is my). Sender was node-1 (received 18 identical messages from node-1 in the last 60010 ms)

                2015-12-24 09:54:04,971 WARN  [org.jgroups.protocols.UDP] (INT-1,my,node-2) JGRP000012: discarded message from different cluster ee (our cluster is my). Sender was node-2 (received 9 identical messages from node-2 in the last 60070 ms)

                2015-12-24 09:54:04,976 WARN  [org.jgroups.protocols.UDP] (INT-1,ee,node-2) JGRP000012: discarded message from different cluster my (our cluster is ee). Sender was node-2 (received 8 identical messages from node-2 in the last 60077 ms)

                 

                But cache data for userCtx cache are not replicated between nodes for SYNC and ASYNC mode.

                I attached logs from both nodes.

                • 5. Re: Wildfly-10.0.0.CR4 replicated-cache issue.
                  Juraj Kubica Newbie

                  Hi. I think, your problem is because of injection of EmbeddedCacheManager. The only way how to inject cache in wildfly 10 is inject Cache directly:

                   

                  @Resource(lookup="java:jboss/infinispan/cache/app_cache/userCtx")

                  Cache userCtx;