8 Replies Latest reply on May 8, 2013 9:10 AM by satyakishor.m

    Issue with modeshape clustering

    satyakishor.m

      I am running modeshape from 2 clusters backed by jdbc infinispan cache. The problem I am running into is that when I add a node into repository from one cluster I am not able to see the new node from other cluster. But when I restart the other cluster I am able to see the new node. Attached my jgroups and repository configuration details. I am running modeshape 3.2 on Jboss eap 6.1 beta server. I also tried with modeshape 3.1.3 on Jboss eap 6.0.1 server and I see the same issue here as well.

       

      Any help is appreciated.

        • 1. Re: Issue with modeshape clustering
          rhauch

          From your description, it sounds like your two servers are not properly clustered. In fact, I'd guess that Infinispan is not clustered properly. Can you please provide your entire AS7/EAP configuration file plus the command you're using to run your servers? The file you supplied is missing a JGroups section.

           

          Oh, you can use ModeShape 3.1.x on AS7.1.1.Final, or ModeShape 3.2 on EAP 6.1.0 (alpha or beta), but no other combinations.

          • 2. Re: Issue with modeshape clustering
            satyakishor.m

            Please find the attached standalone-ha.xml file.

             

            Following are the 2 commands I was using to run jboss in clustered in environment.

             

            standalone.bat -c standalone-ha.xml -b 127.0.0.1 -Djboss.server.base.dir=../standalone-1 -Djboss.node.name=node1 -Djboss.socket.binding.port-offset=100 -Djgroups.bind_addr=127.0.0.1 -Djgroups.bind_port=7600 -Djava.net.preferIPv4Stack=true

            standalone.bat -c standalone-ha.xml -b 127.0.0.1 -Djboss.server.base.dir=../standalone-2 -Djboss.node.name=node2 -Djboss.socket.binding.port-offset=200 -Djgroups.bind_addr=127.0.0.1 -Djgroups.bind_port=7600 -Djava.net.preferIPv4Stack=true

            • 3. Re: Issue with modeshape clustering
              satyakishor.m

              I also see the following messages in the one of the cluster log. It looks like both cluters are talking to each other.

               

              15:29:26,457 INFO  [org.modeshape.jcr.JcrRepository] (MSC service thread 1-2) ModeShape version 3.2.0.Final

              15:29:26,598 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 58) ISPN000078: Starting JGroups Channel

              15:29:26,613 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 59) ISPN000078: Starting JGroups Channel

              15:29:26,660 INFO  [stdout] (ServerService Thread Pool -- 58)

              15:29:26,660 INFO  [stdout] (ServerService Thread Pool -- 58) -------------------------------------------------------------------

              15:29:26,660 INFO  [stdout] (ServerService Thread Pool -- 58) GMS: address=node1/modeshape-index, cluster=modeshape-index, physical address=127.0.0.1:7700

              15:29:26,660 INFO  [stdout] (ServerService Thread Pool -- 58) -------------------------------------------------------------------

               

               

              15:29:26,692 INFO  [stdout] (ServerService Thread Pool -- 59)

              15:29:26,692 INFO  [stdout] (ServerService Thread Pool -- 59) -------------------------------------------------------------------

              15:29:26,692 INFO  [stdout] (ServerService Thread Pool -- 59) GMS: address=node1/modeshape-grid, cluster=modeshape, physical address=127.0.0.1:7700

              15:29:26,692 INFO  [stdout] (ServerService Thread Pool -- 59) -------------------------------------------------------------------

               

               

              15:29:27,082 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 59) ISPN000094: Received new cluster view: [node2/modeshape-grid|1] [node2/modeshape-grid, node1/modeshape-grid]

              15:29:27,129 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 59) ISPN000079: Cache local address is node1/modeshape-grid, physical addresses are [127.0.0.1:7700]

              15:29:27,129 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 59) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.5.Final

              15:29:27,692 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 60) ISPN000031: MBeans were successfully registered to the platform MBean server.

              15:29:27,707 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 58) ISPN000094: Received new cluster view: [node2/modeshape-index|1] [node2/modeshape-index, node1/modeshape-index]

              15:29:27,754 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 58) ISPN000079: Cache local address is node1/modeshape-index, physical addresses are [127.0.0.1:7700]

              • 4. Re: Issue with modeshape clustering
                hchiorean

                Hi,

                 

                A couple of notes regarding the configuration (maybe unrelated to the problem you're seeing):

                 

                1) clustering indexes via Infinispan does not work, unfortunately. See this https://community.jboss.org/message/806785 for more information. This means that if you expect "to see nodes" being replicated using queries, this will not work in a live-system. When server nodes are restarted however, nodes are re-indexed and you'll probably be able to query for them.

                 

                2) when you start up the servers, you use jgroups.bind_port=7600 while the tcp-ms stack has the following initial_hosts defined for TCP_PING: localhost[7750],localhost[7850], with port range=5. I'm no expert in JGroups tuning, but are you sure that the nodes in the ms-cluster cluster see each other ? (I would've expected to also see this in the console output, similar to the nodes in the -grid and -index clusteres, pasted above). I would try (for testing at least) to use as simple a configuration as possible for this stack.

                 

                3) the main repository cache is configured in ASYNC mode. According to the ISPN docs: "With asynchronous replication, errors are simply written to a log. Even when using transactions, a transaction may succeed but replication may not succeed on all cache instances". I'd suggest testing in synchronous mode first, just to make sure there aren't any errors.

                • 5. Re: Issue with modeshape clustering
                  satyakishor.m

                  1) clustering indexes via Infinispan does not work, unfortunately. See this https://community.jboss.org/message/806785 for more information. This means that if you expect "to see nodes" being replicated using queries, this will not work in a live-system. When server nodes are restarted however, nodes are re-indexed and you'll probably be able to query for them.

                           Does this mean that even if the clustering is correctly setup, still we can't search for nodes since there is an issue with clustering indexes via infinispan.

                   

                      

                  2) when you start up the servers, you use jgroups.bind_port=7600 while the tcp-ms stack has the following initial_hosts defined for TCP_PING: localhost[7750],localhost[7850], with port range=5. I'm no expert in JGroups tuning, but are you sure that the nodes in the ms-cluster cluster see each other ? (I would've expected to also see this in the console output, similar to the nodes in the -grid and -index clusteres, pasted above). I would try (for testing at least) to use as simple a configuration as possible for this stack.

                          Can you please share some sample clustering configuration that works so that I will test with that one.

                      

                  3) the main repository cache is configured in ASYNC mode. According to the ISPN docs: "With asynchronous replication, errors are simply written to a log. Even when using transactions, a transaction may succeed but replication may not succeed on all cache instances". I'd suggest testing in synchronous mode first, just to make sure there aren't any errors.

                          I also tried with SYNC mode but I don't see any difference.

                  • 6. Re: Issue with modeshape clustering
                    hchiorean

                    Does this mean that even if the clustering is correctly setup, still we can't search for nodes since there is an issue with clustering indexes via infinispan.

                    This means that if there are multiple nodes (aka. repositories) in the cluster which write/update nodes (which is most likely the case), the indexing results are "unpredictable" at best, or exceptions will be raised in the worst case. The only "safe" way to cluster indexes atm. is to either use JGroups or JMS, which basically ensure that there is only 1 writer node (master).

                     

                    Can you please share some sample clustering configuration that works so that I will test with that one.

                     

                    I don't think we have any EAP/AS7 JGroups configuration example, but the one which we use in the standalone tests is this: https://github.com/ModeShape/modeshape/blob/master/modeshape-jcr/src/test/resources/config/jgroups-test-config.xml. You'd probably need to adapt it to AS7 "syntax".

                    • 7. Re: Issue with modeshape clustering
                      hchiorean

                      I just stumbled across https://issues.jboss.org/browse/MODE-1929. Even though the bug itself is quite trivial, it prevents ModeShape (up-to and including 3.2.0) from clustering (at a repository level) in EAP.

                      What this means, is that nodes created on one cluster-node, even though they are sent across via ISPN, they won't be seen "imediately". In reality, they will appear only when the other cluster-node has finished a "write-operation" on the same workspace.

                      • 8. Re: Issue with modeshape clustering
                        satyakishor.m

                        Glad to know its a bug. I tried with endless configurations without any luck.