6 Replies Latest reply on Sep 26, 2011 6:41 AM by davsclaus

    Concurent Web Services call and design

    johnwalker

      Hi,

       

      Hoping someone can assist me with the following use case:

       

      1) Multiples clients can access a Web Service that is exposing a single operation:

       

      operation(int id, String .. params)

       

      2) Camel must:

       

      1. Validate the client message:

      1a.  The id value must be within a range (between 1 and N)

      1b.  For a given id, there must be M parameters

      For e.g., for id=1: M=3, for id=2: M=3, etc..

      1c.  Log Client requests:

      For e.g.: date, time, operation signature: id, String .. params

      2. Lookup the "target operation" from a database.

      For e.g:

      - id=1 linked to "targetOperation1(String p1, String p2, String p3)"

      - id=2 linked to "targetOperation2(String p1)"

      - etc.

      3. Call the the target operation (external web service)

       

      4. Send the target operation response back to the Client

       

      Questions

       

      1.  How to manage with Camel concurrent call to the web service ?

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

       

      1. Does it create a socket automatically per client request ?

      2. If yes, does it use for that a pool of connections ?

      3. Is concurrency a problem to care about, or does Camel care about it alone ?

      2. Is the following design appropriate ?

      1. Competing Consumer EIP: To parallelize the Clients request and process them independantly from each others

      2. For each paralellized Client request, apply the Request-Reply EIP with a Processor to log, validate, lookup and identify the target Web Service to call

      3. Pipe the target Web Service calls (should/can I do that or should I call the target web service directly?). If so, how do I distinguish the responses and forward them to the correct reply queue ?

      4. Call the target Web Service

      5. Forward back to the reply queue the (request-reply EIP) the target web service response

      6. Send back to the client the response

      3. Is there an exisiting example around treating this problem ?

       

      4. Can Fuse IDE be the right tool to solve such problems ?

       

      Thanks a lot for any help!

      J.

        • 1. Re: Concurent Web Services call and design
          davsclaus

          Ad 1, 2 and 3)

          Camel itself has no web service stack. It integrates with Apache CXF or Spring WebServices. And they leverage a HTTP engine which handles the transport layer. For example Apache CXF supports using an embedded Jetty HTTP engine, or to use a servlet engine from a web server, or to use OSGi HTTP service etc. So its a matter of which runtime you use.

           

          And those HTTP engines support handling concurrent requests. You can often configure/tweak those engines. For example to use a NIO asynchronous messaging, and/or configure thread pool sizes, and whatnot.

           

          All this is taken care of. Often you can just use the default settings.

          • 2. Re: Concurent Web Services call and design
            davsclaus

            Yes Fuse IDE can help you with developing Camel applications. See for example this video of Fuse IDE in action: http://davsclaus.blogspot.com/2011/09/video-using-splitter-eip-and-aggregate.html

             

            There are 2 other videos from links on this page http://fusesource.com/products/fuse-ide/

            • 3. Re: Concurent Web Services call and design
              davsclaus

              In terms of your Camel route, then its something alike

               

              
              <bean id="myValidateBean" class="com.foo.MyValidateBean"/>
              <bean id="lookupTargetBean" class="com.foo.MyLookupTargetBean"></bean>
              
              
              <route>
                 <from uri="cxf:bean:myWebService"></from>
                 <to uri="log:input"></to>    
                 <to uri="bean:myValidateBean?method=validate"></to>
                 <recipientList>
                   <method ref="lookupTargetBean" method="findTarget"></method>
                 </recipientList>
                <to uri="log:output"></to>  
                <!-- when the route ends, the current message is send back as response -->
              </route>
              

               

              See details at

              http://camel.apache.org/how-do-i-use-dynamic-uri-in-to.html

              http://camel.apache.org/recipient-list.html

              http://camel.apache.org/log

              http://camel.apache.org/bean.html

              http://camel.apache.org/bean-binding.html

              • 4. Re: Concurent Web Services call and design
                johnwalker

                Thank You Claus!

                 

                The design I'm thinking of is enclosed. I have been unable to make it with Fuse IDE (where is for e.g. the Polling Consumer ?)

                As you will figure out below, I bought your book! (a great and helpful one!).

                 

                Design

                1. A Client send a message (a CSV File containing lines items)
                             - Either by calling a Web Service
                             - or by uploading a CSV File into a FTP Server

                2. In both case, the mesage goes through a splitter (split per line item)

                3. Get streamed (to lower memory usage)

                4. Get parallelized (to speed up the processing)

                5. Is converted from CSV to Object (to be processed)

                6. Is processed (I guess no way to use a singleton ?)
                             Processing include:
                             - line item validation
                             - logging in database
                             - query database
                             - resulting in message modification

                7. All the parallelized tasks are now aggregated (to avoid the creation of too many request/reply jms queues) and ease the transaction management.

                8. Transaction start: The messages enter in a request-reply jms queues

                9. and call a web service

                Questions

                1. Is this a "good" design ?

                2. How to process multiple files in parallel ?
                       Creating 10 polling consumer to process a bunch of 10 files creating 10 routes ?

                3. How to achieve this with Fuse IDE ?

                4. How will look the route in Java DSL or Spring DSL ?

                That's too many questions. But thank's for your help ! The price to learn Fuse IDE I guess.

                 

                Thanks

                J.

                 

                Edited by: johnwalker on Sep 25, 2011 7:04 PM

                Nice Youtube video! Well explained and well done

                • 5. Re: Concurent Web Services call and design
                  davsclaus

                  The FTP component does not support concurrent consumers, as the underlying FTP framework is not thread safe.

                   

                  We may be able to improve this by adding special support for this in Camel so we can have a poll of FTP worker threads, which can be assigned to download the files in parallel.

                   

                  Instead you can simulate this by just creating X number of routes and link them using direct

                   

                  from ftp1 -> to direct:process

                  from ftp2 -> to direct:process

                  from ftp3 -> to direct:process

                   

                  The trick is you have concurrent ftp clients which will "race for the same files" so you need to use a read lock strategy that works on the FTP server.

                   

                  You can also have 1 ftp route which download the FTP files to a local file system.

                  And the from the file system, you can have concurrent threads

                   

                  from ftp -> to file backup

                   

                  You can then have concurrent consumers using the threads EIP

                  http://camel.apache.org/async

                   

                  And see this blog

                  http://davsclaus.blogspot.com/2009/05/on-road-to-camel-20-concurrency-with.html

                  • 6. Re: Concurent Web Services call and design
                    davsclaus

                    I have created a ticket about the concurrent FTP consumers

                    http://fusesource.com/issues/browse/MR-524