13 Replies Latest reply on Feb 22, 2012 8:03 AM by ivica

    Cache is not replicated on Jboss AS 7.1.2 CR1b

    ivica

      Hi,

       

      I am using two standalone instances of Jboss AS 7.1.2 CR1b, with mod_cluster 1.1.2 on a single Windows 7 machine.

      I am starting the instances with the following commands:

       

      .\bin\standalone.bat -server-config standalone-ha.xml -Djboss.node.name=node1 -b 192.168.1.121

      .\bin\standalone.bat -server-config standalone-ha-2node.xml -Djboss.server.data.dir=standalone\data2 -Djboss.node.name=node2 -b 192.168.1.121

       

      In the standalone-ha-2node.xml, I did defined a different port-binding for the second instance

      <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:100}">

       

      I am using the default Infinispan configuration for my custom cache-container, with the replicated cache policy:

       

      <subsystem xmlns="urn:jboss:domain:infinispan:1.1" default-cache-container="myCacheTest">

                  <cache-container name="myCacheTest" default-cache="repl">

                      <alias>

                          myCacheTest

                      </alias>

                      <transport stack="udp"/>

                      <replicated-cache mode="SYNC" name="repl" batching="true">

                          <locking isolation="REPEATABLE_READ"/>

                      </replicated-cache>

                      <distributed-cache mode="ASYNC" name="dist" batching="true">

                          <file-store/>

                      </distributed-cache>

                  </cache-container>

                  <cache-container name="cluster" default-cache="repl-cluster">

                     ...

                  </cache-container>

                  <cache-container name="web" default-cache="repl">

                     ...

                  </cache-container>

                  <cache-container name="sfsb" default-cache="dist">

                      ...

                  </cache-container>

                  <cache-container name="hibernate" default-cache="local-query">

                      ...

                  </cache-container>

              </subsystem>

       

      I have a demo application that is just persisting values in the DB and doing the basic cache CRUD operations.

       

      Both servers start normally, without any error messages, load balancing over mod_cluster also works normally.

       

      But the cache works as a local-cache, changes are not applied on both nodes. If I start an update operation, it will get persisted in the database and in the cache of one node, but not on the second node.

       

      I also added

      set "JAVA_OPTS=%JAVA_OPTS% -Djava.net.preferIPv4Stack=true"

      in my standalone.conf.bat, but it didn't help.

       

      myCacheTest is recognized by both JBoss instances...

       

      repl-cache.png

       

      I am a newbie in JBoss clustering and Infinispan, and I don't see what am I missing here.

       

      Thanks in Advance

       

      Ivica

        • 1. Re: Cache is not replicated on Jboss AS 7.1.2 CR1b
          galder.zamarreno

          Can you show how your demo application uses the cache?

           

          Also, check the logs to see if the cluster view forms at all.

           

          You might also wanna try setting transport stack name for myCacheTest to 'tcp' in case that helps.

          • 2. Re: Cache is not replicated on Jboss AS 7.1.2 CR1b
            ivica

            Hi,

             

            Here is the demo code

             

            //package and import statements ommited

            @Stateless

            public class OfferManagerBean implements OfferManager {

             

             

                private static final Log _log = LogFactory.getLog(OfferManagerBean.class);

             

             

                @PersistenceContext(unitName = "TEST_PU")

                private EntityManager em ;

               

                      //added the Dependencies: org.infinispan export in the MANIFEST.MF

                      @Resource(lookup="java:jboss/infinispan/myCacheTest")

                private org.infinispan.manager.CacheContainer container;

                private org.infinispan.Cache<Object, Object> cache;

             

             

                @PostConstruct

                public void start() {

                    this.cache = this.container.getCache();

                }

             

                      //I tried out this type of cache startup, also 

                //@Inject

                //private Cache<Object, Object> cache;

             

             

                @Override

                public Offer getOffer(Long id) throws OfferException {

                    String cachedValue = (String)cache.get("offer"+id);

                    if (cachedValue == null) {

                        Query query = em.createNamedQuery(Offer.QUERY_FIND_BY_ID);

                        query.setParameter("_ID", id);

                        Offer offer = (Offer) query.getSingleResult();

                        _log.info("Offer object " + "offer"+offer.getId() + " returned from database");

                        cache.put("offer" + id, offer.getName());

                        _log.info("Offer object " + "offer"+offer.getId() + " is put on cache: " + cache.getName());

                        return offer;

                    } else {

                        Offer offer = new Offer();

                        offer.setId(id);

                        offer.setName(cachedValue);

                        _log.info("Offer object " + cachedValue + " returned from cache");

                        return offer;

                    }

                }

             

             

                @Override

                public Offer update(Offer newOffer) throws OfferException {

                    try {

                        Offer offer = em.find(Offer.class, newOffer.getId());

                        offer.setName(newOffer.getName());

                        em.merge(offer);

                        em.flush();

                        //update offer to cache

                        String cachedValue = (String) cache.get("offer"+offer.getId());

                        if (cachedValue == null) {

                            cache.put("offer"+offer.getId(), offer.getName());

                            _log.info("Offer object " + "offer"+offer.getId() + " created on cache");

                        } else {

                            cache.replace("offer"+offer.getId(), offer.getName());

                            _log.info("Offer object " + "offer"+offer.getId() + " updated on cache");

                        }

                        return newOffer;

                    } catch (Throwable t) {

                        throw new OfferException("Update of offer '"+newOffer.getId()+"' failed: "+t.getMessage());

                    }

                }

            }

             

            I tried the tcp transport stack with the same result

             

            Here is the log from my nodes, with my comments:

             

            ---------------------Node 1----------------------------------------------------------------

            -----------------------------------------------------------------------------------------------

            09:20:30,715 INFO  [org.jboss.web] (MSC service thread 1-1) registering web context: /ponuda

            09:20:30,723 INFO  [org.jboss.as] (MSC service thread 1-1) JBoss AS 7.1.0.CR1b "Flux Capacitor" started in 15484ms - Started 272 of 396 services (117 services are passive or on-demand)

            09:20:30,793 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "ponuda.war"

            09:20:32,072 INFO  [org.jboss.as.clustering.CoreGroupCommunicationService.lifecycle.web] (Incoming-9,null) JBAS010267: New cluster view for partition web (id: 1, delta: 1, merge: false) : [node1/web, node2/web]

            09:20:32,077 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-9,null) ISPN000094: Received new cluster view: [node1/web|1] [node1/web, node2/web]

            09:20:45,539 INFO  [org.jboss.resteasy.cdi.CdiInjectorFactory] (http--192.168.1.121-8080-1) Found BeanManager at java:comp/BeanManager

            09:20:45,674 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (http--192.168.1.121-8080-1) ISPN000078: Starting JGroups Channel

            09:20:45,674 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (http--192.168.1.121-8080-1) ISPN000094: Received new cluster view: [node1/myCacheTest|1] [node1/myCacheTest, node2/myCacheTest]

            09:20:45,675 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (http--192.168.1.121-8080-1) ISPN000079: Cache local address is node1/myCacheTest, physical addresses are [192.168.1.121:55200]

            09:20:45,688 INFO  [org.infinispan.jmx.CacheJmxRegistration] (http--192.168.1.121-8080-1) ISPN000031: MBeans were successfully registered to the platform mbean server.

            09:20:45,690 INFO  [org.jboss.as.clustering] (http--192.168.1.121-8080-1) JBAS010301: Started repl cache from myCacheTest container

            -----------------------COMMENT: calling getOffer methods over load balancer

            09:20:45,834 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8080-1) Offer object offer1 returned from database

            09:20:45,841 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8080-1) Offer object offer1 is put on cache: repl

            09:20:47,617 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8080-1) Offer object Firstt offer returned from cache

            09:20:48,883 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8080-1) Offer object Firstt offer returned from cache

            09:20:50,068 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8080-1) Offer object Firstt offer returned from cache

            09:21:39,337 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8080-1) Offer object Firstt offer returned from cache

            -----------------------COMMENT: calling update method over load balancer

            09:21:58,048 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8080-1) Offer object offer1 updated on cache

            -----------------------COMMENT: calling getOffer methods over load balancer

            09:22:05,540 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8080-1) Offer object The First returned from cache

            09:22:07,400 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8080-1) Offer object The First returned from cache

             

             

            ---------------------------------Node 2--------------------------------------------------------------------------------------

            ----------------------------------------------------------------------------------------------------------------------------------

            09:20:34,871 INFO  [solder-servlet] (MSC service thread 1-2) Catch Integration for Servlets enabled

            09:20:34,882 INFO  [org.jboss.web] (MSC service thread 1-2) registering web context: /ponuda

            09:20:34,940 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "ponuda.war"

            09:20:47,169 INFO  [org.jboss.resteasy.cdi.CdiInjectorFactory] (http--192.168.1.121-8180-1) Found BeanManager at java:comp/BeanManager

            09:20:47,294 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (http--192.168.1.121-8180-1) ISPN000078: Starting JGroups Channel

            09:20:47,294 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (http--192.168.1.121-8180-1) ISPN000094: Received new cluster view: [node1/myCacheTest|1] [node1/myCacheTest, node2/myCacheTest]

            09:20:47,295 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (http--192.168.1.121-8180-1) ISPN000079: Cache local address is node2/myCacheTest, physical addresses are [192.168.1.121:55300]

            09:20:47,314 INFO  [org.infinispan.jmx.CacheJmxRegistration] (http--192.168.1.121-8180-1) ISPN000031: MBeans were successfully registered to the platform mbean server.

            09:20:47,314 INFO  [org.jboss.as.clustering] (http--192.168.1.121-8180-1) JBAS010301: Started repl cache from myCacheTest container

            -----------------------COMMENT: calling getOffer methods over load balancer

            09:20:47,440 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8180-1) Offer object offer1 returned from database

            09:20:47,442 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8180-1) Offer object offer1 is put on cache: repl

            09:20:48,244 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8180-1) Offer object Firstt offer returned from cache

            09:20:49,560 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8180-1) Offer object Firstt offer returned from cache

            09:22:03,847 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8180-1) Offer object Firstt offer returned from cache

            09:22:08,178 INFO  [hr.ponude.dao.OfferManagerBean] (http--192.168.1.121-8180-1) Offer object Firstt offer returned from cache

            • 3. Re: Cache is not replicated on Jboss AS 7.1.2 CR1b
              galder.zamarreno

              I think the problem is in this.container.getCache(); - With that call, you're retrieving the default cache which I don't think it's clustered.

               

              Instead, you should be retrieving one of the named caches that have been configured with a clustered mode, so either, this.container.getCache("repl"); or this.container.getCache("dist");

              • 4. Re: Cache is not replicated on Jboss AS 7.1.2 CR1b
                ivica

                Hi,

                 

                I retrieved cache with

                 

                this.container.getCache("repl");

                 

                but I still have the same problem.

                 

                Is there something I have to configure in the jgroups part?

                • 5. Re: Cache is not replicated on Jboss AS 7.1.2 CR1b
                  galder.zamarreno

                  Hmmm, you shouldn't need to do anything in terms of jgroups and the views look fine.

                   

                  Dunno what's the problem but I'd enable TRACE logging on org.infinispan package, repeat the test and attach the (zipped) logs from both nodes in the cluster.

                  • 6. Re: Cache is not replicated on Jboss AS 7.1.2 CR1b
                    ivica

                    Hi,

                     

                    I've attached the logs.

                     

                    Hope they will be helpful. I didn't find anything that can lead me to a conclusion what is happening.

                     

                    By the way, I reported also another problem in the infinispan quickstart jboss example: https://community.jboss.org/thread/195206?tstart=0

                     

                    Is it possible that this two problems have a common cause.

                     

                    Thanks

                     

                    Ivica

                    • 7. Re: Cache is not replicated on Jboss AS 7.1.2 CR1b
                      galder.zamarreno

                      Hmmm, I still can't see what's wrong. It appears that you're talking to a cache called 'repl' which is marked as a local cache. This is something to do with the AS7 integration layer. Could you repeat the test and on top of the enable TRACE logging, add TRACE for org.jboss.as.clustering package?

                      • 8. Re: Cache is not replicated on Jboss AS 7.1.2 CR1b
                        ivica

                        Hi,

                         

                        I attached logs for tcp and udp transport types (<transport stack="udp"/> and  <transport stack="tcp"/>).

                         

                        This exception is something new:

                         

                        13:25:35,278 TRACE [org.jboss.as.clustering] (MSC service thread 1-5) Could not set FD_SOCK.mcast_addr and FD_SOCK.mcast_port, jgroups-udp-fd socket binding does not specify a multicast socket: java.lang.IllegalStateException: JBAS015301: no multicast binding: jgroups-udp-fd

                                  at org.jboss.as.network.SocketBinding.getMulticastSocketAddress(SocketBinding.java:121) [jboss-as-network-7.1.0.CR1b.jar:7.1.0.CR1b]

                                  at org.jboss.as.clustering.jgroups.JChannelFactory.configureMulticastSocket(JChannelFactory.java:205) [jboss-as-clustering-jgroups-7.1.0.CR1b.jar:7.1.0.CR1b]

                                  at org.jboss.as.clustering.jgroups.JChannelFactory.getProtocolStack(JChannelFactory.java:178) [jboss-as-clustering-jgroups-7.1.0.CR1b.jar:7.1.0.CR1b]

                                  at org.jgroups.JChannel.init(JChannel.java:789) [jgroups-3.0.1.Final.jar:3.0.1.Final]

                                  at org.jgroups.JChannel.<init>(JChannel.java:168) [jgroups-3.0.1.Final.jar:3.0.1.Final]

                                  at org.jboss.as.clustering.jgroups.MuxChannel.<init>(MuxChannel.java:37) [jboss-as-clustering-jgroups-7.1.0.CR1b.jar:7.1.0.CR1b]

                                  at org.jboss.as.clustering.jgroups.JChannelFactory.createChannel(JChannelFactory.java:67) [jboss-as-clustering-jgroups-7.1.0.CR1b.jar:7.1.0.CR1b]

                                  at org.jboss.as.clustering.jgroups.subsystem.ChannelService.start(ChannelService.java:37) [jboss-as-clustering-jgroups-7.1.0.CR1b.jar:7.1.0.CR1b]

                                  at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1824) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]

                                  at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1759) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]

                                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_30]

                                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_30]

                                  at java.lang.Thread.run(Thread.java:662) [:1.6.0_30]

                         

                        But I wasn't able to google something that I can use to solve this problem.

                         

                        Best Regards

                        • 9. Re: Cache is not replicated on Jboss AS 7.1.2 CR1b
                          pferraro


                          Galder Zamarreño wrote:

                           

                          I think the problem is in this.container.getCache(); - With that call, you're retrieving the default cache which I don't think it's clustered.

                           

                          Instead, you should be retrieving one of the named caches that have been configured with a clustered mode, so either, this.container.getCache("repl"); or this.container.getCache("dist");

                          Actually, AS7 overrides the behavior of CacheContainer.getCache() to return the designated default cache (which should be "repl"), so that's not it.

                          • 10. Re: Cache is not replicated on Jboss AS 7.1.2 CR1b
                            pferraro

                            Ivica JBoss User wrote:

                            Both servers start normally, without any error messages, load balancing over mod_cluster also works normally.

                             

                            But the cache works as a local-cache, changes are not applied on both nodes. If I start an update operation, it will get persisted in the database and in the cache of one node, but not on the second node.

                            When you say "the cache works as a local-cache", do you mean the cache mode is LOCAL?  or that your data is not replicating?

                            i.e. in your start() method, what is the value of this.cache.getCacheConfiguration().clustering().cacheMode()?

                             

                            Assuming the cache mode is correct, I see that your cache uses batching, and is therefore transactional.  Perhaps autoCommit is not getting enabled?  What happens if you surround your cache operations with cache.startBatch()/endBatch()?  Try logging the transactional/batching characteristics of your cache in your start() method, so we can see if any of the configuration is getting applied incorrectly.

                            • 11. Re: Cache is not replicated on Jboss AS 7.1.2 CR1b
                              ivica

                              Hi,

                               

                              I mean that my cache is not replicating.

                               

                              This is how the cache is configured in my two standalone-ha configuration files, that are used during the startup:

                               

                                          <cache-container name="myCacheTest" default-cache="repl">

                                              <alias>

                                                  myCacheTest

                                              </alias>

                                              <transport stack="udp"/>

                                              <replicated-cache mode="SYNC" name="repl" batching="true">

                                                  <locking isolation="REPEATABLE_READ"/>

                                              </replicated-cache>

                                              <distributed-cache mode="ASYNC" name="dist" batching="true">

                                                  <file-store/>

                                              </distributed-cache>

                                          </cache-container>

                               

                              I changed the start() method to:

                                  @PostConstruct

                                  public void start() {

                                      this.cache = this.container.getCache(); //tried also with: this.cache = this.container.getCache("repl");

                                      try {

                                          _log.info("Cache mode: " + this.cache.getCacheConfiguration().clustering().cacheMode());

                                          _log.info("Is auto-commit on: " + this.cache.getCacheConfiguration().transaction().autoCommit());

                                          _log.info("batching: " + this.cache.getCacheConfiguration().invocationBatching().enabled());

                                      } catch (Throwable t) {

                                          _log.error("Reading configuration failed: " + t);

                                      }

                                  }

                               

                               

                              And I am getting the following error message:

                              08:32:32,357 ERROR [OfferManagerBean] (http--192.168.1.121-8080-1) Reading configuration failed: java.lang.NoSuchMethodError: org.infinispan.Cache.getCacheConfiguration()Lorg/infinispan/configuration/cache/Configuration;

                               

                              Googling it didn't help much.

                               

                              I have the same exception when I am trying to execute jboss-as7 infinispan quickstart example.

                              • 12. Re: Cache is not replicated on Jboss AS 7.1.2 CR1b
                                pferraro

                                Does your application bundle the infinispan libraries?  It sounds like your application is trying to use a different version of Infinispan than the AS (hence the NoSuchMethodError).

                                • 13. Re: Cache is not replicated on Jboss AS 7.1.2 CR1b
                                  ivica

                                  Yes,

                                   

                                  the reason for this problem was bundling the infinispan libraries. When I changed the scope of the infinispan-core to provided, cache values were replicated on all nodes.

                                   

                                  Thanks for the help.

                                  1 of 1 people found this helpful