3 Replies Latest reply on Feb 6, 2013 10:36 AM by iocanel

Fabric Loadbalancing - tries to access inactive container

pratik_patel Jan 23, 2013 2:55 AM

Hi,

I have configured load balancing using Fuse Fabric. Here is the list of containers and their description. I have attached blueprint files for client and server.

FuseESB:karaf@mtsha1> fabric:container-list

id version alive profiles provision status

mtsha1* 1.0 true fabric, fuse-esb-full, fabric-ensemble-0001-1 success

c11 1.0 false css-port-9090 success

mtsha2 1.0 true fabric, fabric-ensemble-0001-2 success

c12 1.0 true css-port-9090 success

mtsha3 1.0 true fabric, fabric-ensemble-0001-3 success

c13 1.0 true ccc success

FuseESB:karaf@mtsha1>

Container - c11 listens for incoming requests on ip1:9090

Container - c12 listens for incoming requests on ip2:9090

Container - c13 sends the requests to common uri <to uri="fabric-camel:local"/>

The problem is, even if I shutdown one of the server container say c11, the request from client is still trying to access service on c11 and fails with following error.

2013-01-23 06:42:56,700 | WARN | #0 - timer://foo | TimerConsumer | rg.apache.camel.util.CamelLogger 224 | 79 - org.apache.camel.camel-core - 2.10.0.fuse-71-047 | Error processing exchange. Exchange. Caused by: [org.apache.camel.CamelExchangeException - JettyClient failed cause by: Connection refused. Exchange. Caused by: http://java.net.ConnectException - Connection refused]

org.apache.camel.CamelExchangeException: JettyClient failed cause by: Connection refused. Exchange. Caused by: http://java.net.ConnectException - Connection refused

at org.apache.camel.component.jetty.JettyContentExchange.doTaskCompleted(JettyContentExchange.java:214)~~99:org.apache.camel.camel-jetty:2.10.0.fuse-71-047~~

at org.apache.camel.component.jetty.JettyContentExchange.onConnectionFailed(JettyContentExchange.java:147)~~99:org.apache.camel.camel-jetty:2.10.0.fuse-71-047~~

at org.eclipse.jetty.client.HttpExchange$Listener.onConnectionFailed(HttpExchange.java:1096)~~100:org.eclipse.jetty.client:7.6.7.v20120910~~

at org.eclipse.jetty.client.HttpDestination.onConnectionFailed(HttpDestination.java:303)~~100:org.eclipse.jetty.client:7.6.7.v20120910~~

at org.eclipse.jetty.client.SelectConnector.startConnection(SelectConnector.java:103)~~100:org.eclipse.jetty.client:7.6.7.v20120910~~

at org.eclipse.jetty.client.HttpDestination.startNewConnection(HttpDestination.java:277)~~100:org.eclipse.jetty.client:7.6.7.v20120910~~

at org.eclipse.jetty.client.HttpDestination.doSend(HttpDestination.java:561)~~100:org.eclipse.jetty.client:7.6.7.v20120910~~

at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:499)~~100:org.eclipse.jetty.client:7.6.7.v20120910~~

at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:165)~~100:org.eclipse.jetty.client:7.6.7.v20120910~~

at org.apache.camel.component.jetty.JettyHttpProducer.doSendExchange(JettyHttpProducer.java:250)~~99:org.apache.camel.camel-jetty:2.10.0.fuse-71-047~~

at org.apache.camel.component.jetty.JettyHttpProducer.process(JettyHttpProducer.java:94)~~99:org.apache.camel.camel-jetty:2.10.0.fuse-71-047~~

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.component.jetty.JettyHttpProducer.process(JettyHttpProducer.java:88)~~99:org.apache.camel.camel-jetty:2.10.0.fuse-71-047~~

at org.fusesource.fabric.camel.FabricLocatorEndpoint$3.process(FabricLocatorEndpoint.java:111)~~103:org.fusesource.fabric.fabric-camel:7.1.0.fuse-047~~

at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:44)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:71)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.fusesource.fabric.camel.FabricLocatorEndpoint$1.process(FabricLocatorEndpoint.java:59)~~103:org.fusesource.fabric.fabric-camel:7.1.0.fuse-047~~

at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:122)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:298)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:117)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.fabric.FabricTraceProcessor.process(FabricTraceProcessor.java:81)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:334)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:135)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:63)~~79:org.apache.camel.camel-core:2.10.0.fuse-71-047~~

at java.util.TimerThread.mainLoop(Timer.java:512)~~:1.6.0_20~~

at java.util.TimerThread.run(Timer.java:462)~~:1.6.0_20~~

Caused by: java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)~~:1.6.0_20~~

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)~~:1.6.0_20~~

at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)~~:1.6.0_20~~

at org.eclipse.jetty.client.SelectConnector.startConnection(SelectConnector.java:79)~~100:org.eclipse.jetty.client:7.6.7.v20120910~~

... 57 more

Shouldn't the requests go to only active container?

Statistics from client

root@c13> camel:route-info fabric-client

Camel Route fabric-client

Camel Context: camel

Properties

id = fabric-client

parent = fc02c8

Statistics

Exchanges Total: 33274

Exchanges Completed: 32891

Exchanges Failed: 383

Min Processing Time: 3ms

Max Processing Time: 30648ms

Mean Processing Time: 15ms

Total Processing Time: 494076ms

Last Processing Time: 473ms

Load Avg: 0.13, 0.12, 0.12

First Exchange Date: 2013-01-22 12:33:42

Last Exchange Completed Date: 2013-01-23 07:02:47

Definition

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<simple>Hello from Fabric Client to group "Local"</simple>

</setBody>

<log message=">>> $ : ${header.karaf.name}" id="log1"/>

</route>

Edited by: pratik_patel on Jan 23, 2013 1:25 PM

client blueprint.xml 1.0 KB
server blueprint.xml 1.4 KB

1. Re: Fabric Loadbalancing - tries to access inactive container

pratik_patel Jan 23, 2013 6:15 AM (in response to pratik_patel)

When I shutdown the container, I do not see associated entries from zookeeper registry and when I start container, the entries are displayed.
So it looks like zookeeper registry is correctly updated. If so, why does client still tries to send a request of inactive container? Am I missing anything?

( BTW, it shows two entries for the same container when container is active. not sure if this is expected behavior )

zk:list -r -d | grep 9090
-
fabric/registry/camel/endpoints/local/00000000030 = jetty:http://192.168.6.211:9090/fabric
fabric/registry/camel/endpoints/local/00000000031 = jetty:http://192.168.6.211:9090/fabric
fabric/registry/camel/endpoints/local/00000000016 = jetty:http://192.168.6.212:9090/fabric
fabric/registry/camel/endpoints/local/00000000017 = jetty:http://192.168.6.212:9090/fabric

Thanks,
Pratik

Edited by: pratik_patel on Jan 23, 2013 4:44 PM
Actions
2. Re: Fabric Loadbalancing - tries to access inactive container

iocanel Jan 23, 2013 7:08 AM (in response to pratik_patel)

I logged an issue to investigate: http://fusesource.com/issues/browse/FABRIC-359
Actions
3. Re: Fabric Loadbalancing - tries to access inactive container

iocanel Feb 6, 2013 10:36 AM (in response to pratik_patel)

Added a fix that addresses this issue. It should be available in the next release.
Actions

Go to original post