-
1. Re: Concurent Web Services call and design
davsclaus Sep 25, 2011 10:39 AM (in response to johnwalker)Ad 1, 2 and 3)
Camel itself has no web service stack. It integrates with Apache CXF or Spring WebServices. And they leverage a HTTP engine which handles the transport layer. For example Apache CXF supports using an embedded Jetty HTTP engine, or to use a servlet engine from a web server, or to use OSGi HTTP service etc. So its a matter of which runtime you use.
And those HTTP engines support handling concurrent requests. You can often configure/tweak those engines. For example to use a NIO asynchronous messaging, and/or configure thread pool sizes, and whatnot.
All this is taken care of. Often you can just use the default settings.
-
2. Re: Concurent Web Services call and design
davsclaus Sep 25, 2011 10:40 AM (in response to johnwalker)Yes Fuse IDE can help you with developing Camel applications. See for example this video of Fuse IDE in action: http://davsclaus.blogspot.com/2011/09/video-using-splitter-eip-and-aggregate.html
There are 2 other videos from links on this page http://fusesource.com/products/fuse-ide/
-
3. Re: Concurent Web Services call and design
davsclaus Sep 25, 2011 10:48 AM (in response to johnwalker)In terms of your Camel route, then its something alike
<bean id="myValidateBean" class="com.foo.MyValidateBean"/> <bean id="lookupTargetBean" class="com.foo.MyLookupTargetBean"></bean> <route> <from uri="cxf:bean:myWebService"></from> <to uri="log:input"></to> <to uri="bean:myValidateBean?method=validate"></to> <recipientList> <method ref="lookupTargetBean" method="findTarget"></method> </recipientList> <to uri="log:output"></to> <!-- when the route ends, the current message is send back as response --> </route>
See details at
http://camel.apache.org/how-do-i-use-dynamic-uri-in-to.html
http://camel.apache.org/recipient-list.html
-
4. Re: Concurent Web Services call and design
johnwalker Sep 25, 2011 3:10 PM (in response to davsclaus)Thank You Claus!
The design I'm thinking of is enclosed. I have been unable to make it with Fuse IDE (where is for e.g. the Polling Consumer ?)
As you will figure out below, I bought your book! (a great and helpful one!).
Design
A Client send a message (a CSV File containing lines items)
- Either by calling a Web Service
- or by uploading a CSV File into a FTP ServerIn both case, the mesage goes through a splitter (split per line item)
Get streamed (to lower memory usage)
Get parallelized (to speed up the processing)
Is converted from CSV to Object (to be processed)
Is processed (I guess no way to use a singleton ?)
Processing include:
- line item validation
- logging in database
- query database
- resulting in message modificationAll the parallelized tasks are now aggregated (to avoid the creation of too many request/reply jms queues) and ease the transaction management.
Transaction start: The messages enter in a request-reply jms queues
and call a web service
Questions
Is this a "good" design ?
How to process multiple files in parallel ?
Creating 10 polling consumer to process a bunch of 10 files creating 10 routes ?How to achieve this with Fuse IDE ?
How will look the route in Java DSL or Spring DSL ?
That's too many questions. But thank's for your help ! The price to learn Fuse IDE I guess.
Thanks
J.
Edited by: johnwalker on Sep 25, 2011 7:04 PM
Nice Youtube video! Well explained and well done
-
design.png 123.3 KB
-
5. Re: Concurent Web Services call and design
davsclaus Sep 26, 2011 6:39 AM (in response to johnwalker)The FTP component does not support concurrent consumers, as the underlying FTP framework is not thread safe.
We may be able to improve this by adding special support for this in Camel so we can have a poll of FTP worker threads, which can be assigned to download the files in parallel.
Instead you can simulate this by just creating X number of routes and link them using direct
from ftp1 -> to direct:process
from ftp2 -> to direct:process
from ftp3 -> to direct:process
The trick is you have concurrent ftp clients which will "race for the same files" so you need to use a read lock strategy that works on the FTP server.
You can also have 1 ftp route which download the FTP files to a local file system.
And the from the file system, you can have concurrent threads
from ftp -> to file backup
You can then have concurrent consumers using the threads EIP
And see this blog
http://davsclaus.blogspot.com/2009/05/on-road-to-camel-20-concurrency-with.html
-
6. Re: Concurent Web Services call and design
davsclaus Sep 26, 2011 6:41 AM (in response to davsclaus)I have created a ticket about the concurrent FTP consumers