4 Replies Latest reply on Mar 1, 2013 2:40 PM by joseotavio

    JBoss AS 7 Cluster / Infinispan Issue

    rschamm

      Hi all,

       

      I have been playing with JBoss AS 7 clustering the last few weeks, and in particular the role of infinispan in clustering.

       

      My test env. is setup more or less as described in this article:

       

      https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto

       

      I use JBoss AS 7 domain mode, with one "master" host and one "slave" host. Further I use the "full-ha" profile for the "other-server-group".

       

      ...

      <server-group name="other-server-group" profile="full-ha">

      ...

       

      So in the "master" host I start server "server-three":

       

      ...

      <server name="server-three" group="other-server-group" auto-start="true">

      ...

       

      And in the "slave" host I start server "server-six" (I renamed the three slave servers to 4,5 and 6):

       

      ...

      <server name="server-six" group="other-server-group" auto-start="true">

      ...

       

      I successfully tested web session replication, sfsb replication and the second-level hibernate cache with this setup.

       

      But when I try and test this bean to use the infinispan "cluster" cache container for my own purposes I start running into issues:

       

      ...

      @ConcurrencyManagement(ConcurrencyManagementType.CONTAINER)

      @Singleton

      public class CacheBean

      {

         @Resource(lookup = "java:jboss/infinispan/container/cluster")

         private org.infinispan.manager.CacheContainer container;

         private org.infinispan.Cache<String, String> cache;

       

         public CacheBean()

         {

         }

        

         @PostConstruct

         public void start()

         {

            System.out.println("*********************");     

            System.out.println("Init cache!!!!");

            System.out.println("*********************");     

            this.cache = this.container.getCache();

         }

       

       

         @Lock(LockType.WRITE)

         public void save(String t)

         {

            cache.put(generateKey(), t);

         }

       

       

         @Lock(LockType.WRITE)

         public void clear()

         {

            cache.clear();

         }

       

       

         @Lock(LockType.READ)

         public List<String> getCacheList()

         {

            List<String> dataList = new ArrayList<String>();

            dataList.addAll(cache.values());

            return dataList;

         }

       

       

         private String generateKey()

         {

            String uuid = UUID.randomUUID().toString();

            return uuid;

         }

      }

      ...

       

       

      Everything works fine if I start up "master" first and then "slave", and then first use the cache against the master instance. So I wait for the "master" to finish starting up and then start up the "slave". Then I call a servlet using the above bean against the master instance and then I call the same servlet in either "slave" or "master" and everything works fine.

       

      But, if I first use the cache (e.g in a servlet) against the "slave" the page locks up for 60 seconds, and then I see this log in the slave output:

       

      org.infinispan.util.concurrent.TimeoutException: Timed out after 60 seconds waiting for a response from master:server-three/cluster

      org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:333)

      org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:132)

      org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:492)

      org.infinispan.cacheviews.CacheViewsManagerImpl.join(CacheViewsManagerImpl.java:213)

       

      Has anyone see this kind of output before: "Timed out after 60 seconds waiting for a response from master:server-three/cluster" ?

       

      I hope I made some sense. Everything else I tested so far works like a charm, but this last issue is still eluding me.

       

      Thanks all!

        • 1. Re: JBoss AS 7 Cluster / Infinispan Issue
          galder.zamarreno

          What AS7 version are you using?

          • 2. Re: JBoss AS 7 Cluster / Infinispan Issue
            pferraro

            How is your application referencing the Infinispan classes?

            * Have you defined an explicit module dependency in your MANIFEST.MF? (good)

            * Did you include the infinispan jar in your deployment archive? (bad)

             

            Just to say, in AS 7.1, you can inject the Cache directly via @Resource(lookup = "java:jboss/infinispan/cache/cluster/default")

            • 3. Re: JBoss AS 7 Cluster / Infinispan Issue
              rschamm

              Hi guys thanks for trying to help. Just wanted to let you know that I managed to get rid of the issue entirely by starting the cache eagerly (start="EAGER").

              • 4. Re: JBoss AS 7 Cluster / Infinispan Issue
                joseotavio

                I have been using Infinispan in JBoss 7.1.3 over the past weeks to replicate information accross the nodes of my cluster. A few weeks ago I ran into the issue described in https://community.jboss.org/thread/218920 and managed to fix it by configuring only my cache containers (and not the caches) to start eagerly.

                 

                However, recently I started getting the error described in this thread. I would start my first server with no problem and then start the second one, but when I tried to execute an operation on the second server which required access to a cache which had not yet been started in the first server, I would get this timeout exception. If I execute the same operation on the first server the cache gets started there, and from there on I can execute the operation on the second server too with no problems.

                 

                So I decided to start all the caches eagerly (as suggested above) to make a test and I started having the first issue again: the first server loads fine, but the second one can't load the caches because it tries to synchronize them before loading my application modules, therefore the classes of the objects in the caches are only being loaded after Infinispan tries to sync the caches.

                 

                I think right now the problem is restricted to only one of my caches, so I can get around it by starting only this one cache eagerly (this way I don't get the timeout error nor the sync error), but I'm afraid I might run into the same issue in some other case in the future, so I have a question: is there any way to make sure the caches will only start and sync after my modules have been loaded?