Load Balancing Question

[djharte]

Hi Folks,

Hopefully this is an easy one; I have just started to try out a few things with Switchyard/Camel and am super interested in understanding how load balancing is configured. I have followed the cluster setup instructions and am running two servers (on one Windows machine) with standalone-ha configuration. I have deployed a number of services one of which is marked as remote ( <remote:binding.remote/>). So, my use case is as follows: I want to split a list into sublists and send the sublists to be processed by the different (clustered) instances of the same service.My routing code is as follows:

from("switchyard://Process").routeId("${body}").to("switchyard://EntitySelectionModel")

        .split(ExpressionBuilder.beanExpression(

                  new MyEntityIteratorFactory(),  "createIterator" )).loadBalance().roundRobin().to("switchyard://ProfitModel")

The ProfitModel is the remote service, I was hoping that the sublists would be split across the two instances of the ProfitModel service. Unfortunately though they are all being routed to the "node1" server.

Any advice on how to get this to work?

Thanks,

David

[kcbabo]

Hey David,

There are actually two layers of load balancing possible here and it's probably best to go with one. 

First, The loadBalance() EIP should be used when you want to load balance across multiple services.  IOW, you have multiple references defined in your switchyard.xml and you want to load balance across them like this:

loadBalance().roundRobin().to("switchyard://ProfitModel1", "switchyard://ProfitModel2")

Where ProfitModel1 and ProfitModel2 are both defined as references in your application.

The second option is to use the built-in load balancing support inside our remote binding.  So in your case, the ProfitModel would be promoted with a binding.remote binding.  If you had two nodes with a service called "ProfitModel" deployed with a binding.remote binding, then the remote client will load balance between them internally.

~ keith

[djharte]

Thanks very much for replying Keith. The thing is, I really don't want to manage multiple definitions of the same service, although maybe I haven't throught this through enough. So, if I understand correctly you are saying that I don't need to call loadBalance().roundRobin().to("switchyard://ProfitModel1") as Switchyard will take care of this for me if I make the service remote ( via  'remote.binding' which I am currently using). If this is the case, how do I change the behavior of the load balancing algorithm?

I'll test it out later when I get home

Thanks again,

David

[kcbabo]

If your aim is to just load balance across multiple nodes in a SY cluster, then I would recommend using the load balancing built into the remote invoker (option 2 in my last reply).  There is an API hook for configuring the load balancing behavior, but no configuration hook.  The good news is that it defaults to round robin. :-)  We're going to spend some time in 0.8 tightening up the invoker and clustering components in the runtime and configuration area.

[djharte]

I must have something amiss with my cluster or the remote setup although I think I have correctly followed the instructions. My service is defined as follows:

		<sca:component name="ProfitModelBean">
			<bean:implementation.bean class="com.example.switchyard.profitmodel.ProfitModelBean" />
			<sca:service name="ProfitModel">
				<sca:interface.java interface="com.example.switchyard.profitmodel.ProfitModel" />
				<remote:binding.remote />
			</sca:service>
		</sca:component>

and the reference:

		<sca:component name="PrePositioningBean">
			<camel:implementation.camel>
				<camel:java class="com.example.switchyard.prepositioning.PrePositioningRoute" />
			</camel:implementation.camel>
			<sca:service name="PrePositioning">
				<sca:interface.java interface="com.example.switchyard.prepositioning.PrePositioning" />
			</sca:service>
			<sca:reference name="ProfitModel">
				<sca:interface.java interface="com.example.switchyard.profitmodel.ProfitModel" />
				<remote:binding.remote />
			</sca:reference>
			<sca:reference name="EntitySelectionModel">
				<sca:interface.java interface="com.example.switchyard.entityselectionmodel.EntitySelectionModel" />
			</sca:reference>
		</sca:component>

The output from the cluster appearst to be correct and I can call the service individually on each port, 8080 and 9080 (offset is 1000), it's a REST service by the way. But the output is not load balanced.

As suggested, I removed the Camel load balancing EIP, giving:

		from("switchyard://PrePositioning").routeId("${body}").to("switchyard://EntitySelectionModel")
		.split(ExpressionBuilder.beanExpression(
				new MyEntityIteratorFactory(),  "createIterator" )).to("switchyard://ProfitModel");

My HTTP call is: http://localhost:(8/9)080/rest-binding/prepos/test

Any ideas?

Thanks,

David

[kcbabo]

The binding.remote for the reference needs to be on the composite reference and not the component reference.  We should yell a bit louder in the deployer when we encounter this situation, so I've added a JIRA:

https://issues.jboss.org/browse/SWITCHYARD-1291

Aside from the above, can you give me a quick idea of your deployment architecture here?  If you are deploying the same application on two different instances, then the reference is never going to resolve to the non-local (second node) endpoint.  The idea behind that decision is (a) it's significantly slower to go out of process and (b) it doesn't add to availability because if the first node goes down there's not routing to perform anyway.

[djharte]

Hi Keith,

The use case I am trying to solve is one of scalability; when the ProfitModel processes a set of entities it needs to perform work which can be resource expensive. The real world use case is financial, a set of Entities is made available to the system for which trading profit needs to be calculated. The profit calculation requires retrieval of historical trades (for a period), reference and market data for each Entity and then performs the profit calculation. Distributing the entities over a number of instances of the same (ProfitModel) service would allow the process to be scaled and promote parallel computation.

Also, I haven't tried this yet, but I was thinking of just sending through the entitiy ids and storing any calculated state in Infinispan (the rest of the data is transient).

Do you think this is a valid use case for Switchyard/Camel?

Thanks,

David

[kcbabo]

Hey David,

That use case is totally valid and sounds like a great type of example that we might want to include as an example at some point. :-)  My question about deployment architecture was actually a bit more literal though - I'm curious how many SY applications you are using in this use case and how they are deployed across a cluster.  Is it the same app deployed three different times?  Or is it a single "feeder" app which does the splitting and routing and another "worker" app which is distributed across clustered nodes?  In the feeder/worker scenario, the worker apps would contain the expensive processing logic and scale out on multiple nodes to improve performance.   If it's the same application deployed in three separate nodes, then you are likely hitting a design constraint of our current addressing handler - it will route to services provided inside an application (i.e. local) in preference to services provided outside the application.  The idea here is that a local invocation is faster than a remote invocation in many cases.  If the service being invoked is particularly expensive, then it might make more sense to balance out the requests.

cheers,

keith

[djharte]

Hi Keith,

Thanks for the response and apologies for not replying sooner (I'm on vacation at the moment). I think it makes sense to follow a deployment strategy similar to the feeder/worker scenario you described, in this topology I would expect the feeder process to comprise the Camel orchestration logic where each of the end points (ProfitModel (and others)) would be a remote service/separate SY application. Can you confirm please that the default load balancing behavior would be leveraged in this case?

One question I have though, would I need a common project for the shared POJO classes being updated by each of the applications (i.e. Entity) or is there a better way of doing this in SY?

I'll put together a diagram of what I think the topology should look like once I get back just to confirm I understand correctly.

Thanks again,

David

[djharte]

Hi Keith,

Quick follow up question: I am attempting to do as you suggested; created a feeder (SY) application and a separate worker (SY) app and am running into an issue to get the two talking. I have setup a clustered environment as outlined in the docs and have deployed the worker service (ProfitModel) as a BeanService with remote binding. I am attempting to access this service via the following Camel sequence:

from("switchyard://PrePositioning").routeId("${body}").to("switchyard://EntitySelectionModel")

        .split(ExpressionBuilder.beanExpression(

                  new MyEntityIteratorFactory(),  "createIterator" )).to("switchyard://ProfitModel");

But am getting the error:

00:05:00,545 ERROR [org.apache.camel.processor.DefaultErrorHandler] (http--127.0.0.1-8080-1) Failed delivery for (MessageId: ID-Veracruz-54812-1361328

843688-106-7 on ExchangeId: ID-Veracruz-54812-1361328843688-106-9). Exhausted after delivery attempt: 1 caught: java.lang.NullPointerException: No Ser

viceReference was found for uri [switchyard://ProfitModel?namespace=urn%3Aswitchyard-example%3Aresteasy-binding%3A1.0]: java.lang.NullPointerException

: No ServiceReference was found for uri [switchyard://ProfitModel?namespace=urn%3Aswitchyard-example%3Aresteasy-binding%3A1.0]

        at org.switchyard.component.camel.SwitchYardProducer.lookupServiceReference(SwitchYardProducer.java:110) [switchyard-component-camel-switchyar

However, I see the service listed in the console and the uri details appear to be the same leading me to believe I am doing something wrong (by the way the service has only one method). Can you therefore please confirm that in a clustered environment the two applications should be able to communicate as I have described or do I need to add a binding to facilitate communication?

Thanks,

David

[djharte]

... ok, so I think the issue was the way I had my cluster setup. However I know have a Camel versioning NoSuchMethodError to contend with (on node 2). Can you confirm please which version of Camel I should be running, currently I am using the bundled version: 2.10.0.

Thanks

David

.1-10080-1) Servlet.service() for servlet SwitchYardRemotingServlet threw exception: java.lang.RuntimeException: java.lang.NoSuchMethodException: org.

apache.camel.processor.Splitter.<init>()

        at org.switchyard.common.type.reflect.Construction.construct(Construction.java:166) [switchyard-common-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.common.type.reflect.Construction.construct(Construction.java:71) [switchyard-common-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.DefaultFactory.create(DefaultFactory.java:37) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.node.AccessNode.decompose(AccessNode.java:133) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.Graph.decomposeReference(Graph.java:150) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.node.MapNode.decompose(MapNode.java:80) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.Graph.decomposeReference(Graph.java:150) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.node.AccessNode$1.run(AccessNode.java:145) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.Graph.decomposeRoot(Graph.java:136) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.GraphSerializer.deserialize(GraphSerializer.java:66) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.component.remote.SwitchYardRemotingServlet.doPost(SwitchYardRemotingServlet.java:75) [switchyard-component-remote-0.7.0.Fina

l.jar:0.7.0.Final]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]

        at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0]

Caused by: java.lang.NoSuchMethodException: org.apache.camel.processor.Splitter.<init>()

        at java.lang.Class.getConstructor0(Class.java:2721) [rt.jar:1.7.0]

        at java.lang.Class.getDeclaredConstructor(Class.java:2002) [rt.jar:1.7.0]

        at org.switchyard.common.type.reflect.Construction.construct(Construction.java:161) [switchyard-common-0.7.0.Final.jar:0.7.0.Final]

        ... 24 more

23:10:16,142 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/switchyard-remote].[SwitchYardRemotingServlet]] (http--127.0.0

.1-10080-1) Servlet.service() for servlet SwitchYardRemotingServlet threw exception: java.lang.RuntimeException: java.lang.NoSuchMethodException: org.

apache.camel.processor.Splitter.<init>()

        at org.switchyard.common.type.reflect.Construction.construct(Construction.java:166) [switchyard-common-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.common.type.reflect.Construction.construct(Construction.java:71) [switchyard-common-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.DefaultFactory.create(DefaultFactory.java:37) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.node.AccessNode.decompose(AccessNode.java:133) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.Graph.decomposeReference(Graph.java:150) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.node.MapNode.decompose(MapNode.java:80) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.Graph.decomposeReference(Graph.java:150) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.node.AccessNode$1.run(AccessNode.java:145) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.Graph.decomposeRoot(Graph.java:136) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.serial.graph.GraphSerializer.deserialize(GraphSerializer.java:66) [switchyard-serial-0.7.0.Final.jar:0.7.0.Final]

        at org.switchyard.component.remote.SwitchYardRemotingServlet.doPost(SwitchYardRemotingServlet.java:75) [switchyard-component-remote-0.7.0.Fina

l.jar:0.7.0.Final]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]

        at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0]

[kcbabo]

Can you confirm please that the default load balancing behavior would be leveraged in this case?

Yes, if there are multiple instances of an application deployed in the cluster with the same service then load balancing will direct traffic between them with a round robin as a default.

One question I have though, would I need a common project for the shared POJO classes being updated by each of the applications (i.e. Entity) or is there a better way of doing this in SY?

I consider it a best practice to externalize shared classes, particularly when you look to employ governance to track these things when more and more applications begin to use them.  From a clustering standpoint, we serialize the payload as JSON so you should not run into a class compatibility issue if you decide to package the class(es) with your applications instead of externalizing them into a distinct module.

Something to keep in mind here is that the clustering config is changing slightly in 0.8.  It will be more flexible in terms of namespace requirements and will also support introducing custom load balance strategies.  Discussion of the change can be found here:

https://community.jboss.org/message/798228#798228

Complete docs on the latest clustering support and an example app / quickstart will be available with 0.8.0.Final.

[kcbabo]

David Harte wrote:

... ok, so I think the issue was the way I had my cluster setup. However I know have a Camel versioning NoSuchMethodError to contend with (on node 2). Can you confirm please which version of Camel I should be running, currently I am using the bundled version: 2.10.0.

I think this may be an issue with deserializing a Camel context property.  Any chance you can attach a copy of your latest app that's being deployed to node 2? 

[djharte]

Hi Keith,

I have attached both apps, the 2nd on is called switchyard-profitability.

Once I get this realtively simple use case to work i'll check out the new features in 0.8.0.

Thanks,

David

[dward]

Thanks for making us aware of this scenario, David.  I've created a jira for the fix work: SWITCHYARD-1325.  Your example app will be useful in testing the fix.