Camel Integration

[talawahdotnet]

Hi,

I was wondering if you guys had laid out your plans yet for how camel integration will work.  I really like the fresh approach being taken with this project but I am also very wary of the fact that it might be another 12 - 18 months before we see a GA product. In the meantime I was hoping to use a Camel + HornetQ combo to tackle our current integration needs and I figured that since Switchyard's ESB core is based on HornetQ it may be possible for me essentially deploy camel on top of HornetQ to create a light-weight standalone integration platform while Switchyard development continues.

Is what I am suggesting feasible? Would it be similiar to what is already planned for Switchyard? Or should I just scrap that approach and deploy camel to a JBoss EAP instance that already has HornetQ running?

Any suggestions or corrections are welcome.

[kcbabo]

Hey,

HornetQ core will be a bus provider inside of SwitchYard.  The current thinking is that there will be different providers based on the needs and policy of the application.  For example, if you have two services local to your application that can interact in a synchronous manner and exchange messages by reference, then an InVM style provider would be best for that.  If you need persistence, transactions, scalability to large producer/consumer populations, then HornetQ is best.  I hacked up some initial stuff around this last week, but have not had a chance to clean it up and publish it in a topic branch in my repo.  Give me 3 days or so and I should be able to provide a bit more info on the provider SPI and how HornetQ will fit into it.

As far as Camel and HornetQ go, this combination will be available in SwitchYard.  We plan to have a Camel component which allows users to define pipeline routes between services hosted in SwitchYard.  The camel routes ideally just route between SwitchYard services, which abstracts out the binding of the service implementation.  The actual messages exchange on the route could travel through HornetQ, the in-memory provider, or a number of other possibilities.  This should be a policy/configuration detail and not something cooked into the endpoint definition in a Camel configuration, IMHO.

In terms of timing, I'm hoping to have an initial/rough implementation useable with HornetQ in the next two weeks.  Camel is on tap for Milestone 2 of SwitchYard, which should kick off in about two weeks time as well.  We would definitely be interested in feedback as we travel this path.

cheers,

keith

[talawahdotnet]

Hmmm, based on how you have described it, you seem to be talking about using Camel from the perspective of routing/orchestration between endpoints defined in Switchyard.  Is that Camel's primary role in Switchyard?  Do you guys plan to make use of the multitude of available Camel components for connecting to external endpoints as well? Would the overall syntax be similar to what exists in Camel now or will the use of Camel based routing be somewhat independent of the use of Camel connectors?

I am just getting started with a simple integration project so I was looking at things much more from a data routing/transformation perspective than a "Services" perspective.  The grand SOA vision is further down ther road, but right now I just want to move/transform/filter messages.  For example, how would Sitchyard support a simplified workflow where I just wanted to route XML files from one directory to another, filtering out those that don't match a given criteria defined in xpath? What does it look like when I want to switch from local directories to SFTP/JMS/Amazon SQS?  Is it going to be something that can be specified fairly easily/briefly like in camel or would I have to write code to define a Service for each endpoint?

I like the approach of treating the bus provider as configurable component that can easily be changed depending on your needs.  This will make the process of moving from proof of concept to robust, reliable system that much easier.  Similarly, it would be great if the definition of routes and the corresponding enpoints could make the move from POC to production just as seamless.

[kcbabo]

Hmmm, based on how you have described it, you seem to be talking about using Camel from the perspective of routing/orchestration between endpoints defined in Switchyard.  Is that Camel's primary role in Switchyard?  Do you guys plan to make use of the multitude of available Camel components for connecting to external endpoints as well? Would the overall syntax be similar to what exists in Camel now or will the use of Camel based routing be somewhat independent of the use of Camel connectors?

The first aspect of Camel integration we will be looking at is for routing and pipeline orchestration.  Camel wll be wrapped and exposed as an implementation component in SwitchYard, which will allow you to define and include a Camel route inside a SwitchYard application to coordinate a "flow" between services.  In this context, Camel is basically a composition engine.  Other options for service composition include a CDI bean (multiple services can be invoked via @Reference injection), jBPM, BPEL, etc.   I think Camel has done a great job of defining a clear syntax for the routing rules, so I believe that it will be the primary mechanism folks use when defining routes/pipelines in SwitchYard.

Camel also has a large selection of components/adapters which could also be used in SwitchYard to handle service and reference binding (i.e. gateways).  It's not the first thing we are looking at right now, but it's definitely something we have discussed and will be looking at after we take care of the routing integration.

I am just getting started with a simple integration project so I was looking at things much more from a data routing/transformation perspective than a "Services" perspective.  The grand SOA vision is further down ther road, but right now I just want to move/transform/filter messages.  For example, how would Sitchyard support a simplified workflow where I just wanted to route XML files from one directory to another, filtering out those that don't match a given criteria defined in xpath? What does it look like when I want to switch from local directories to SFTP/JMS/Amazon SQS?  Is it going to be something that can be specified fairly easily/briefly like in camel or would I have to write code to define a Service for each endpoint?

You raise a very interesting point here w/r/t SOA and integration.  In my experience, a great number of projects start by solving the integration problem first and then look at service-orientation.  This is certainly true for existing EAI-type environments where SOA is being bolted on top of an existing solution, but it also happens in new projects as well.  I think the best way to address it is to make it easy to solve EAI/integration problems inside of a service-oriented framework, which is what we are trying to do in SwitchYard.

Using your example above, you need to make a decision about the services in your application.  If there are no services, then SwitchYard is not bringing a lot to the table.  That said, I can see a number of angles to how the transform/filter application should contain a "service".  For example, what are the expected data types entering the filter/transform route?  Will the entity pushing files need to know ahead of time?  If the types change, is this significant to the client or the transformation logic?  Will the functionality be exposed over multiple protocols?  Would you want to invoke the service from a business process without wiring in protocol-binding details (e.g. the directory where you need to write a file)? 

Looking at this from a SwitchYard perspective, you could create a simple Camel route as a service within SwitchYard.  The from: of the route would be an internal dispatcher, which is mapped via configuration to a gateway binding.  Any transforms, filters, etc. can execute as part of the Camel route and the final to: could be the name of a SwitchYard service.  The destination service could be mapped to any protocol via a reference binding in the SwitchYard configuration.  In this case, you are still using Camel to solve the integration problem, but doing it in SwitchYard allows you to do it in a service-oriented manner.

I like the approach of treating the bus provider as configurable component that can easily be changed depending on your needs.  This will make the process of moving from proof of concept to robust, reliable system that much easier.  Similarly, it would be great if the definition of routes and the corresponding enpoints could make the move from POC to production just as seamless.

I got stuck on deployment-related stuff most of last week, so I'm a bit behind on posting my HornetQ provider prototype.  The good news is that I finished off the deployer stuff last night, so I should be able to take care of HornetQ this week for sure. ;-)

[kcbabo]

Looking at this from a SwitchYard perspective, you could create a simple Camel route as a service within SwitchYard.  The from: of the route would be an internal dispatcher, which is mapped via configuration to a gateway binding.  Any transforms, filters, etc. can execute as part of the Camel route and the final to: could be the name of a SwitchYard service.  The destination service could be mapped to any protocol via a reference binding in the SwitchYard configuration.  In this case, you are still using Camel to solve the integration problem, but doing it in SwitchYard allows you to do it in a service-oriented manner.

One point of clarification on my comments above.  You don't have to treat every Camel endpoint as a service, although I would consider it a best practice.  Doing so allows you to establish a clear set of dependencies for your service.  However, if a particular endpoint dependency is a private detail of your service implementation, you are not forced to expose that as a service.  It's up to the developer of the application to determine if/how concrete endpoint dependencies impact the flexibility of their service(s).

[talawahdotnet]

Keith Babo wrote:
Using your example above, you need to make a decision about the services in your application.  If there are no services, then SwitchYard is not bringing a lot to the table.  

Well this first thing I see SwitchYard bringing to the table is the HornetQ based bus provider.  Being able to add that in seamlessly is a big win in my mind.

The evolution of my application would look something like this:

1) Start out with a simple Camel route for moving messages from a file based endpoint to a JMS based endpoint.  If SwitchYard could transparently generate the gateway binding configurations based on the "from" and "to" in my route great.

2) Add a filter to the pipeline to drop certain messages using Camel's built in filters

3) Turn on the HornetQ bus provider so that messages are persisted even if one of the endpoints goes away.

Once that part is up and working I would probably start thinking about wrapping the "to" endpoint with a simple service interface that accepts my custom Java object format (MyMessage). I would of course define a Transformer to allow SwitchYard to convert from the message body to a MyMessage object. This service would

1) Validate the message content according to certain rules

2) Call a logging service to log events

3) Send off an email alert if the outgoing JMS Gateway is down.

But aside from those utility features, all the "service" does is push messages. Failure responses (even validation failures) would always be handled in a separate process since both the upstream and downstream endpoints are InOnly.

Keith Babo wrote:
One point of clarification on my comments above.  You don't have to treat every Camel endpoint as a service, although I would consider it a best practice.  Doing so allows you to establish a clear set of dependencies for your service.  However, if a particular endpoint dependency is a private detail of your service implementation, you are not forced to expose that as a service.  It's up to the developer of the application to determine if/how concrete endpoint dependencies impact the flexibility of their service(s).

I guess I'll have to wait and see what the workflow ends up looking like to get a full appreciation. I am just hoping that the "message routing first, services later" approach is sufficiently intuitive so that folks who aren't looking at the problem from a purely service driven perspective aren't immediately daunted; especially those coming from a Camel background.

[kcbabo]

I'm already talking with another community member about creating a Camel component for our current (0.1) release.  I agree that the service driven approach cannot be daunting.  The idea is to add a relatively small amount of overhead that will pay big dividends down the line. 

I think the workflow you defined above for defining a route is reasonable and I will capture it as a use case for our Camel integration.  Hopefully, we can crank out an initial implementation quickly and you can test it out and let us know what you think.

cheers,

keith

[talawahdotnet]

Sounds like a plan.