-
1. Re: Issue with modeshape clustering
rhauch May 2, 2013 4:57 PM (in response to satyakishor.m)From your description, it sounds like your two servers are not properly clustered. In fact, I'd guess that Infinispan is not clustered properly. Can you please provide your entire AS7/EAP configuration file plus the command you're using to run your servers? The file you supplied is missing a JGroups section.
Oh, you can use ModeShape 3.1.x on AS7.1.1.Final, or ModeShape 3.2 on EAP 6.1.0 (alpha or beta), but no other combinations.
-
2. Re: Issue with modeshape clustering
satyakishor.m May 2, 2013 5:03 PM (in response to rhauch)Please find the attached standalone-ha.xml file.
Following are the 2 commands I was using to run jboss in clustered in environment.
standalone.bat -c standalone-ha.xml -b 127.0.0.1 -Djboss.server.base.dir=../standalone-1 -Djboss.node.name=node1 -Djboss.socket.binding.port-offset=100 -Djgroups.bind_addr=127.0.0.1 -Djgroups.bind_port=7600 -Djava.net.preferIPv4Stack=true
standalone.bat -c standalone-ha.xml -b 127.0.0.1 -Djboss.server.base.dir=../standalone-2 -Djboss.node.name=node2 -Djboss.socket.binding.port-offset=200 -Djgroups.bind_addr=127.0.0.1 -Djgroups.bind_port=7600 -Djava.net.preferIPv4Stack=true
-
3. Re: Issue with modeshape clustering
satyakishor.m May 2, 2013 5:19 PM (in response to rhauch)I also see the following messages in the one of the cluster log. It looks like both cluters are talking to each other.
15:29:26,457 INFO [org.modeshape.jcr.JcrRepository] (MSC service thread 1-2) ModeShape version 3.2.0.Final
15:29:26,598 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 58) ISPN000078: Starting JGroups Channel
15:29:26,613 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 59) ISPN000078: Starting JGroups Channel
15:29:26,660 INFO [stdout] (ServerService Thread Pool -- 58)
15:29:26,660 INFO [stdout] (ServerService Thread Pool -- 58) -------------------------------------------------------------------
15:29:26,660 INFO [stdout] (ServerService Thread Pool -- 58) GMS: address=node1/modeshape-index, cluster=modeshape-index, physical address=127.0.0.1:7700
15:29:26,660 INFO [stdout] (ServerService Thread Pool -- 58) -------------------------------------------------------------------
15:29:26,692 INFO [stdout] (ServerService Thread Pool -- 59)
15:29:26,692 INFO [stdout] (ServerService Thread Pool -- 59) -------------------------------------------------------------------
15:29:26,692 INFO [stdout] (ServerService Thread Pool -- 59) GMS: address=node1/modeshape-grid, cluster=modeshape, physical address=127.0.0.1:7700
15:29:26,692 INFO [stdout] (ServerService Thread Pool -- 59) -------------------------------------------------------------------
15:29:27,082 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 59) ISPN000094: Received new cluster view: [node2/modeshape-grid|1] [node2/modeshape-grid, node1/modeshape-grid]
15:29:27,129 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 59) ISPN000079: Cache local address is node1/modeshape-grid, physical addresses are [127.0.0.1:7700]
15:29:27,129 INFO [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 59) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.5.Final
15:29:27,692 INFO [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 60) ISPN000031: MBeans were successfully registered to the platform MBean server.
15:29:27,707 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 58) ISPN000094: Received new cluster view: [node2/modeshape-index|1] [node2/modeshape-index, node1/modeshape-index]
15:29:27,754 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 58) ISPN000079: Cache local address is node1/modeshape-index, physical addresses are [127.0.0.1:7700]
-
4. Re: Issue with modeshape clustering
hchiorean May 3, 2013 2:03 AM (in response to satyakishor.m)Hi,
A couple of notes regarding the configuration (maybe unrelated to the problem you're seeing):
1) clustering indexes via Infinispan does not work, unfortunately. See this https://community.jboss.org/message/806785 for more information. This means that if you expect "to see nodes" being replicated using queries, this will not work in a live-system. When server nodes are restarted however, nodes are re-indexed and you'll probably be able to query for them.
2) when you start up the servers, you use jgroups.bind_port=7600 while the tcp-ms stack has the following initial_hosts defined for TCP_PING: localhost[7750],localhost[7850], with port range=5. I'm no expert in JGroups tuning, but are you sure that the nodes in the ms-cluster cluster see each other ? (I would've expected to also see this in the console output, similar to the nodes in the -grid and -index clusteres, pasted above). I would try (for testing at least) to use as simple a configuration as possible for this stack.
3) the main repository cache is configured in ASYNC mode. According to the ISPN docs: "With asynchronous replication, errors are simply written to a log. Even when using transactions, a transaction may succeed but replication may not succeed on all cache instances". I'd suggest testing in synchronous mode first, just to make sure there aren't any errors.
-
5. Re: Issue with modeshape clustering
satyakishor.m May 3, 2013 10:47 AM (in response to hchiorean)1) clustering indexes via Infinispan does not work, unfortunately. See this https://community.jboss.org/message/806785 for more information. This means that if you expect "to see nodes" being replicated using queries, this will not work in a live-system. When server nodes are restarted however, nodes are re-indexed and you'll probably be able to query for them.
Does this mean that even if the clustering is correctly setup, still we can't search for nodes since there is an issue with clustering indexes via infinispan.
2) when you start up the servers, you use jgroups.bind_port=7600 while the tcp-ms stack has the following initial_hosts defined for TCP_PING: localhost[7750],localhost[7850], with port range=5. I'm no expert in JGroups tuning, but are you sure that the nodes in the ms-cluster cluster see each other ? (I would've expected to also see this in the console output, similar to the nodes in the -grid and -index clusteres, pasted above). I would try (for testing at least) to use as simple a configuration as possible for this stack.
Can you please share some sample clustering configuration that works so that I will test with that one.
3) the main repository cache is configured in ASYNC mode. According to the ISPN docs: "With asynchronous replication, errors are simply written to a log. Even when using transactions, a transaction may succeed but replication may not succeed on all cache instances". I'd suggest testing in synchronous mode first, just to make sure there aren't any errors.
I also tried with SYNC mode but I don't see any difference.
-
6. Re: Issue with modeshape clustering
hchiorean May 3, 2013 10:59 AM (in response to satyakishor.m)Does this mean that even if the clustering is correctly setup, still we can't search for nodes since there is an issue with clustering indexes via infinispan.
This means that if there are multiple nodes (aka. repositories) in the cluster which write/update nodes (which is most likely the case), the indexing results are "unpredictable" at best, or exceptions will be raised in the worst case. The only "safe" way to cluster indexes atm. is to either use JGroups or JMS, which basically ensure that there is only 1 writer node (master).
Can you please share some sample clustering configuration that works so that I will test with that one.
I don't think we have any EAP/AS7 JGroups configuration example, but the one which we use in the standalone tests is this: https://github.com/ModeShape/modeshape/blob/master/modeshape-jcr/src/test/resources/config/jgroups-test-config.xml. You'd probably need to adapt it to AS7 "syntax".
-
7. Re: Issue with modeshape clustering
hchiorean May 8, 2013 8:13 AM (in response to hchiorean)I just stumbled across https://issues.jboss.org/browse/MODE-1929. Even though the bug itself is quite trivial, it prevents ModeShape (up-to and including 3.2.0) from clustering (at a repository level) in EAP.
What this means, is that nodes created on one cluster-node, even though they are sent across via ISPN, they won't be seen "imediately". In reality, they will appear only when the other cluster-node has finished a "write-operation" on the same workspace.
-
8. Re: Issue with modeshape clustering
satyakishor.m May 8, 2013 9:10 AM (in response to hchiorean)Glad to know its a bug. I tried with endless configurations without any luck.