Multinode Remote Cluster with 5.3Beta2 Servers

[kalinwilson]

Tristan, et al,

First, I'm enjoying getting to know Infinispan and I appreciate the documentation on the wiki, although some of the older pages are conflicting with newer documentation. However, I'm surprised there isn't more explicit documentation on setting up a remote cluster/grid, in either a test or operational environment. I expect that this is a very prominent use case for an IMDG, vice local caches clustered by way of multiple process instances, which seems to be the model most covered. Even in Manik's Packt book, the remote case is not covered in any detail. I'm using the wiki page at https://docs.jboss.org/author/display/ISPN/Infinispan+Server, which I see was recently added/updated. It's good about explaining how a single server instance is configured, but doesn't go into how to run several server instances in a cluster (unless I missed it). I'm not familiar with JBOSS AS yet, so I haven't tried to figure out AS clustering.

I'm using bin/clustered.sh to try to set up a 2+ node server cluster on a single machine. When I start the second server I get errors as the protocols try to bind to the same ports the first server is using. I tried using both standalone/configuration/clustered.xml and docs/examples/configs/clustered-two-nodes.xml. Neither would let me simply start multiple instances of the servers, which is what I expect I have to do to get multiple nodes. I looked at the JBOSS AS command line parameters but did not see how to bind hotrod to a different port.

I think I follow the AS configuration for the caches and associated subsystems. Do I need to create config file variants that change the protocol bindings in the <socket-binding-group> element> and use a different file for each node?

*** Would you explicitly lay out the steps for establishing a multinode remote cluster both on a single machine and different machines? Adding this to the wiki would be helpful for a lot of people I bet.

Once I have the cluster up, I expect I can write a client. How do I connect to the cluster with the command line client? I tried ./ispn-cli.sh --connect=//kalin@localhost:11222/clustered/default to connect to the hotrod port with no luck. I also tried jmx://localhost:12000 per the instructions in https://docs.jboss.org/author/display/ISPN/Command-Line+Interface+%28CLI%29.

Additionally, is there a way to remotely inspect the contents of each node of a remote cluster to verify distribution, etc.? Via JMX or the CLI?

I looked at the differences between clustered.xml and clustered-two-nodes.xml and some of the other config files. They seem to be minor. The removal of the modcluster extension, different unicast protocols, the addition of the rsvp protocol, etc. Could you explain how each example config is supposed to be used? A readme in the distro would help a lot.

My evaluation of Infinispan has gone well so far but I really need to verify how a remote cluster is configured, managed, and used. Sorry for the long post but Thanks for any help!

Kalin

[mgencur]

Hello Wilson,

first, I won't be able to answer all your questions but let me answer just some of them.

Running two clustered Infinispan servers on one host is as easy as passing a port-offset system property when starting the second server:

./clustered.sh -Djboss.socket.binding.port-offset=100

I was looking at the https://docs.jboss.org/author/display/ISPN/Infinispan+Server and it contains quite a lot of useful information. Sure, it still needs to be extended to cover more configuration elements. On the other hand, it also contains a link to the JBossAS page describing command line parameters, which is good to follow when you want to deploy ISPN server in production environment.

You're right that the CLI documentation is a bit outdated. There have been changes recently, where the ISPN server was based on JBossAS. And the documentation was not updated since then. Connecting to a running instance of ISPN server via CLI:

./ispn-cli.sh --connect=remoting://user:password@localhost:9999/clustered/default or

./ispn-cli.sh --connect=localhost

(now you need to connect over 9999 which is a port for the management interface)

wrt JMX/CLI - you can connect to the server and verify that the cluster was formed, you can also retrieve values for the cluster via CLI but you cannot get the list of all keys in each node

You can also check how to connect to a running server via HotRod, Memcached or REST interface in the examples here: https://github.com/infinispan/jdg-quickstart (esp.  the endpoint examples)

Martin

[kalinwilson]

Thanks Martin. This should help me get a little further.

Are you familiar with how the Infinispan team tests remote cluster nodes to verify distribution hashing, etc? Do they attach debuggers to each node or is there anything else, except maybe logging?

If you're an Infinispan user, do you have expereience with remote clusters? What has your experience been with getting started and testing?

Thanks again,

Kalin

[mgencur]

Hello Kalin,

as you know, the server has been recently re-based on JBossAS so the testsuite structure changed a bit. There are still tests in ISPN core which test server endpoints (not with replicated caches, AFAIK). There are also other tests in the server's testsuite (https://github.com/infinispan/infinispan-server/tree/master/testsuite) which test example configuration files. I don't think there are debuggers attached anywhere in the tests. The tests run automatically and, correct me if I'm wrong, the debuggers need interactive process. The remote caches can be examined via JMX to some extent. logs are the other option.

I have quite a bit of experience with the remote servers, both getting them started and testing them ;-) If you have further questions, pls start another discussion.

Martin

[nadirx]

Kalin, the tests are not done via debuggers, but through carefully designed unit tests which start up clusters of node and verify that data gets actually distributed in a repeatable manner (and we have a continuous integration server which runs the testsuite at every commit).

The tests are at https://github.com/infinispan/infinispan/tree/master/core/src/test/java/org/infinispan/distribution

I've updated the Infinispan Server Getting Started section to include answers to your questions above. Please let me know what you think

[kalinwilson]

Thanks alot for the documentation update, Tristan. I think that will help a lot. I'll take a look at the test suite. I'm sure that will help too.

Thanks also to Martin for the test suite tip and the offer for future help.

Kalin