please help test clustering

[kcbabo]

Code still needs to be cleaned up a bit (more than a bit in some places), but it's functional and nothing in terms of configuration or setup should change between now and beta 2.  If you have a few spare cycles, please bang on it real quick and see how quickly the wheels fall off the cart.

First thing you need to do is pull from my parent, core, components, and release branches and run a build.  There's a small issue with the AS7 extension where standalone is broken (you need to use standalone-ha).  I will sort that out as part of my clean-up.  For now, build release with "mvn -Dmaven.test.skip clean install".

git pull git://github.com/kcbabo/parent.git SWITCHYARD-730

git pull git://github.com/kcbabo/core.git SWITCHYARD-730

git pull git://github.com/kcbabo/components.git SWITCHYARD-730

git pull git://github.com/kcbabo/release.git SY-730   (note the branch name change)

After that, unzip the built switchyard-as7.zip into two directories (e.g. node1, node2).  Start each one using the following configuration:

node1 : "bin/standalone.sh -c standalone-ha.xml"

node2 : "bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node2 -Djboss.socket.binding.port-offset=2000"

You can setup more instances if you like, but be sure to provide a unique node name and port offset for each additional instance.

Once the servers are running, build and deploy the attached apps.  Deploy app1 to node1 and app2 to node2.  Then go into the app1 project and run the following to bounce a message off the instance:

mvn exec:java

That will send a test message to node1, which will find the service in node2 and invoke it.  The return value is passed back to node1 and then back to the original invoker.

Hopefully, all of the above will work for you. :-)  If it doesn't, let me know.  If it does, see if you can try some alternate scenarios like taking a quickstart and breaking it across two nodes.  Anything like that will help out.

thanks,

keith

[cwash]

Worked for me.  I set up everything on a dinky Windows laptop that I have, things seemed to work well. (After Maven downloaded the Internet...!)

Noticed in your notes last pull needs to be on release:

git pull git://github.com/kcbabo/release.git SY-730   (note the branch name change)

Also, the release build failed complaining about the remote component not being installed... I just went in and installed it manually and that seemed to do the trick.  Not sure if the build picked it up as a submodule - wasn't paying super close attention.

Got the following output --

client:

[INFO] --- exec-maven-plugin:1.2.1.jbossorg-3:java (default-cli) @ app1 ---
02:07:25,431 INFO  [mixins.HTTPMixIn] Sending a POST request to [http://localhost:8080/app1]
02:07:25,462 INFO  [mixins.HTTPMixIn] Request body:[keith]
02:07:25,587 INFO  [mixins.HTTPMixIn] Received response header:[Server=Apache-Coyote/1.1]
02:07:25,587 INFO  [mixins.HTTPMixIn] Received response header:[Content-Length=22]
02:07:25,587 INFO  [mixins.HTTPMixIn] Received response header:[Date=Sat, 27 Oct 2012 06:07:25 GMT]
02:07:25,587 INFO  [mixins.HTTPMixIn] Received response body:[Hey keith, Adios dude!]

node1

02:07:25,540 INFO  [stdout] (http--127.0.0.1-8080-1) GreetingBean ::: Hello keith

node2

02:07:25,571 INFO  [stdout] (http--127.0.0.1-10080-2) Hey keith, Adios dude!

[igarashitm]

failover works!

node1+app1, node2+app2, node3+app2

Pattern A

1. start node1, node2, node3
2. invoke [node1 => node2]
3. shutdown node2
4. invoke [node1 => node3]
5. shutdown node3
6. invoke [node1 =>(N/A)]
7. start node2
8. invoke [node1 => node2]
9. start node3
10. invoke [node1 => node2]

Pattern B

1. start node1, node3, node2
2. invoke [node1 => node2]
3. shutdown node2
4. invoke [node1 => node3]
5. start node2
6. invoke [node1 => node2]

node2 seems to be given a priority. Do we have any plan for load balancing?

And do we have a Warning on node1 if no endpoint is available? (Pattern A / Step 6)

Thanks,

Tomo

[kcbabo]

Chris - thanks for testing it out and for identifying a typo in the instructions.  The release repo URL has been fixed in the original post.

[kcbabo]

Thanks, Tomo.  There is no load balancing planned in 0.6, but for 0.7 we will support it via a endpoint selection strategy in the remote invoker.  Added a JIRA for tracking purposes:

https://issues.jboss.org/browse/SWITCHYARD-1141

Re: step 6 of Pattern A, the outbound path in the remote component throws a HandlerException when no remote endpoints can be found.  This will be returned as a fault and since the MEP for this invocation is IN-ONLY, the fault should be logged.  My guess would be that the logging is not happening for some reason.  Need to investigate that in core and bus-camel.

[kcbabo]

FYI - I tested this across multiple machines with no issues.

[kkopalle]

Hi Keith,

Thanks for the excellent examples which worked for me. I did a simple load test by modifying app1 client as follows and I am getting a very low throughput of 250 service invocations per sec. Is that expected? I am running everything (three AS7 instances and the client program) on a single computer.

Regards,

Kishore

Modified client:

package com.example.switchyard.app1;

import org.switchyard.test.mixins.HTTPMixIn;

public class Client {

          private static long ITER = 10000;

    public static void main(String[] args) throws Exception {

        final HTTPMixIn http = new HTTPMixIn();

        http.initialize();

        http.setDumpMessages(false);

                    long startTime = System.nanoTime();

                    for(int i = 0; i < ITER; i++)

                    {

                              Thread thread = new Thread()

                              {

                                        public void run()

                                        {

                                                  http.postString("http://localhost:8080/app1", "Kishore");

                                        }

                              };

                              thread.start();

                              thread.join();

                    }

                    System.out.println("It took " + ((System.nanoTime()-startTime)/1e9) + " seconds for " + ITER + " iterations");

    }

}

[kcbabo]

Sorry, I completely missed this post.  250 sounds very low.  I'll play around with it this week and see what kind of numbers I get on my machine.

thanks,

keith

[kcbabo]

OK, I'm back after a bit of playing.  Before we get to actual numbers, I want to discredit anything I share here based on a few factors:

1) These are sample apps, not real apps.  In the case of my cluster test, the implementation logic is ridiculously small.

2) These are "out of the box" numbers with zero tuning

3) I'm quite possibly the world's worst load tester.  A fact which I realized this morning to my dismay.

With that out of the way, I was able to submit 4000 messages to a cluster app which I created in 6162ms (tps = 649).  Some differences from your test client:

- multiple test submission threads (I tried up to 8)

- removed System.out.println and logging (this didn't have huge impact)

The performance of our internal routing protocol can be improved, so I would expect to see a modest bump in that number once we get to that.

For sake of comparison, I tried another application without clustering to see what I could get.  I chose the bean-service quickstart from our distribution, deployed it on a single instance and tested with the load client in soapUI.  The result of that was 43163 messages in 30 seconds (tps = 1437).  Again, no tuning of the runtime.  Another thing to keep in mind is that this particular example uses JAXB for transformation for the request *and* response message.

As we approach SOA-P 6 release, we'll get more performance testing from our QE team which will lead to more performance improvements and tuning advice in the runtime.

hth,

keith