7 Replies Latest reply on Oct 31, 2011 2:46 PM by Keith Babo

    Camel implementation (single and shared environment)

    Michael Burman Newbie

      Moving from IRC to here to get some opinions. Idea comes from the issue we had with JBossESB's Camel implementation and seeing that SwitchYard employs the same issues, I thought it might I'll give a suggestion. The use-case here in short is that some services are used by several customers, whom send their data for example to the front-end SFTP server and then the data is fetched from the front-end to the backend (safe DMZ deployment for front-end). In this case, there are a lot of Camel inbounds and adding new customer should happen on the fly. The current implementation would require restart & redeploy, since everything is defined in the XML files right now. That's also not very convenient for production to handle.

       

      Since CamelContext allows modification of Routes to be added/modified/removed dynamically, wouldn't it make sense to allow SwitchYard services to use these features also? In other words, expose the route handling for example through JMX (just an example, not a huge fan of JMX). It would allow adding new camel bindings from a user interface such as webapp.

       

      That would also mean that they would have to be stored in some repository to be queryable. Here's when I come to my second proposition. Earlier in the project there was an idea to share the data between SwitchYard instances using Infinispan. That would serve two the first point of modifications to the routes as well as provide a safeguard in which servers handle certain Camel routes (protocols). Assume that we would have two machines sharing the same service with same Camel inbounds. If there would be file gateways or FTP gateways, then that could cause duplicate messages being routed to the service. So shared datastore could be at the same time used to communicate the status to other nodes, who would be responsible for polling these inbound ports.

       

      The high-availability / scalable SwitchYard is of course another issue, but its got relevance to the Camel implementation.

        • 1. Re: Camel implementation (single and shared environment)
          Randall Hauch Master

          Has there been any thought or discussion of using a JCR repository and in particular, ModeShape? As with other JCR implementations, you can manage all kinds of structured content and metadata, and even query the content using multiple languages, including a SQL-like language called JCR-SQL2. But ModeShape offers a few things that other JCR implementations don't.

           

          With ModeShape, you can upload a file and have a custom ModeShape sequencer automatically parse the file, extract the meaningful structure, and store that structure into the repository where it can be queried and/or navigated just like any other content. Our Reference Guide describes how sequencers work, and includes examples of sequencers that are included out of the box, like the XSD sequencer and WSDL sequencers.

           

          Also, ModeShape works really well with Infinispan, and can even store its content inside an Infinispan cache. This capability will be expanded even more with ModeShape 3.0, which is just now getting started (see our forum soon for discussions about this).

          • 2. Re: Camel implementation (single and shared environment)
            Michael Burman Newbie

            Alright, my point wasn't actually about choosing the datastore that's behind the scenes. It was more about that there should be one that's available distributed to other SW instances and would be used to store this sort of data. The interface to this data could hide the actual implementation (and have just some default one that JBoss feels is good for the 'whole product catalog' or something, I'm sure you have some plans how to integrate different products to create nice end-user experience). And the other issue was that I wanted some sort of views on how people would think this should be solved (pointing to "lightweight container" information).

            • 3. Re: Camel implementation (single and shared environment)
              Randall Hauch Master

              Gotcha. Didn't mean to step on your discussion. But if you continue to go in this direction, please keep ModeShape in mind.

              • 4. Re: Camel implementation (single and shared environment)
                Keith Babo Master

                Moving from IRC to here to get some opinions. Idea comes from the issue we had with JBossESB's Camel implementation and seeing that SwitchYard employs the same issues, I thought it might I'll give a suggestion. The use-case here in short is that some services are used by several customers, whom send their data for example to the front-end SFTP server and then the data is fetched from the front-end to the backend (safe DMZ deployment for front-end). In this case, there are a lot of Camel inbounds and adding new customer should happen on the fly. The current implementation would require restart & redeploy, since everything is defined in the XML files right now. That's also not very convenient for production to handle.

                 

                Only if you deploy everything inside a single application.  You can deploy your service implementation (routing logic, etc.) as an independent application and augment that with other deployments which contain additional bindings for the service.

                 

                Since CamelContext allows modification of Routes to be added/modified/removed dynamically, wouldn't it make sense to allow SwitchYard services to use these features also? In other words, expose the route handling for example through JMX (just an example, not a huge fan of JMX). It would allow adding new camel bindings from a user interface such as webapp.

                 

                The from: endpoint in a Camel route is a SwitchYard service, which makes the route independent of any specific gateway binding.  So dynamically updating the route definition is not going to allow you to add bindings.

                 

                That would also mean that they would have to be stored in some repository to be queryable. Here's when I come to my second proposition. Earlier in the project there was an idea to share the data between SwitchYard instances using Infinispan. That would serve two the first point of modifications to the routes as well as provide a safeguard in which servers handle certain Camel routes (protocols). Assume that we would have two machines sharing the same service with same Camel inbounds. If there would be file gateways or FTP gateways, then that could cause duplicate messages being routed to the service. So shared datastore could be at the same time used to communicate the status to other nodes, who would be responsible for polling these inbound ports.

                 

                The high-availability / scalable SwitchYard is of course another issue, but its got relevance to the Camel implementation.

                 

                The goal here is to allow service components (modular bits of your application) to be stored in a repository.  You can take these bits and package them into an application which contains the binding and configuration relevant to your environment.  We have also discussed whether application properties can be altered at runtime.  I think this could be a useful feature and allows configuration details of an application to be externalized and manageable outside of the development and packaging process - so at deployment and runtime.

                • 5. Re: Camel implementation (single and shared environment)
                  Keith Babo Master

                  Hey Randall - our initial exploration here has been with the new generified Guvnor.  Since ModeShape can run inside Guvnor, I would be interested in exploring how we can use sequencers and some of the other useful features of ModeShape under the hood.

                  Has there been any thought or discussion of using a JCR repository and in particular, ModeShape? As with other JCR implementations, you can manage all kinds of structured content and metadata, and even query the content using multiple languages, including a SQL-like language called JCR-SQL2. But ModeShape offers a few things that other JCR implementations don't.

                  • 6. Re: Camel implementation (single and shared environment)
                    Michael Burman Newbie

                    Keith Babo wrote:

                     

                    Only if you deploy everything inside a single application.  You can deploy your service implementation (routing logic, etc.) as an independent application and augment that with other deployments which contain additional bindings for the service.

                     

                    Pretty irrelevant to the original problem though, no matter if they're in the same application or not. Otherwise one would end up deploying thousands of applications each one with one Camel route. Sounds like a management nightmare

                     

                    The idea was to not change the production environment itself at all. But instead let people such as sellers to configure the systems while selling and just printing out the form "here's your password to our SFTP server, your service is activated". Altering production is always risky in my opinion, even if only deploying new stuff (who knows what that might break).

                     

                     

                    The from: endpoint in a Camel route is a SwitchYard service, which makes the route independent of any specific gateway binding.  So dynamically updating the route definition is not going to allow you to add bindings.

                     

                    Alright, so how should they be added then? Yes and I do believe this is about the issue of changing application properties at runtime. And I also believe it's a requirement that shouldn't be overlooked, including just shutting down polling inputs / receiving answers etc. We do it all the time when we know telecom guys are going to "fix" something..

                    • 7. Re: Camel implementation (single and shared environment)
                      Keith Babo Master

                      Michael Burman wrote:

                       

                      Keith Babo wrote:

                       

                      Only if you deploy everything inside a single application.  You can deploy your service implementation (routing logic, etc.) as an independent application and augment that with other deployments which contain additional bindings for the service.

                       

                      Pretty irrelevant to the original problem though, no matter if they're in the same application or not. Otherwise one would end up deploying thousands of applications each one with one Camel route. Sounds like a management nightmare

                       

                      The idea was to not change the production environment itself at all. But instead let people such as sellers to configure the systems while selling and just printing out the form "here's your password to our SFTP server, your service is activated". Altering production is always risky in my opinion, even if only deploying new stuff (who knows what that might break).

                       

                      The user is not adding a new Camel route each time - they are simply deploying a new gateway binding (which is a Camel component) for an existing service.  If the requirement is that you need to add bindings to an existing service, then the solution I presented will address that requirement.  It's true that this starts to get difficult to manage if you have 1000s of applications where each one requires it's own specific binding, but that's going to be true anyway you slice it.  I'm not sure how dynamically adding endpoints resolves the management issue - adding might be "easier", but changing or removing could be tough to track.  Either of these activities on a Camel route would require a restart of the route, so that's another management issue with dynamically altering a single route.

                       

                      Alright, so how should they be added then? Yes and I do believe this is about the issue of changing application properties at runtime. And I also believe it's a requirement that shouldn't be overlooked, including just shutting down polling inputs / receiving answers etc. We do it all the time when we know telecom guys are going to "fix" something..

                       

                      SCA provides a standard way to provide properties that can be specified outside of a service component or binding definition.  We don't have support for this at the moment, but it's something we do plan on providing.  Here's a JIRA to track it:

                       

                      https://issues.jboss.org/browse/SWITCHYARD-520