1 2 Previous Next 22 Replies Latest reply on Mar 18, 2007 10:22 PM by Ronald van Kuijk

    Embedded FTP Server and FTP Clients

    Bill Burke Master

      A couple of things:

      I have integrated Apache's FTP Server into an embeddable SAR file. It supports FTP and FTP over SSL and numerous ways to plug in user database as well as FTPLets. For starters, I'd like to get permission to commit this and integrate it with the testsuite so that the testsuite can run out of the box rather than you having to manually start your own FTP server on your box, or having to use somebody elses.

      Another thing I think we can do with this FTP server, is write an FTPLet so that we can push Files to the ESB rather than having to poll it. I don't know the FTPLet api enough yet, but I imagine it is even possible to hook into the stream as the remote client is ftpping the file. This would allow the ESB to process FTP messages entirely in memory, quickly, and have all the efficiency of a stream.

      Speaking of streaming, one of the difficiencies of the FTP listener is that it downloads the FTP file to the ESB's machine before processing the file. This seems extremely inefficient and instead again, maybe we should offer streaming capability there as well. The problem is the FTP Client we are using only supports downloading the file. You can't "get" an InputStream. I looked around a bit and the Jakarta Commons Net library allows you to get by InputStream.

        • 1. Re: Embedded FTP Server and FTP Clients
          Kurt Stam Master

          +1 from me for adding to the embeddable SAR. But please wait till more people respond. I think it'd be a great improvement.

          It's nice that we could revert to pushing, but this would be something we should consider after MP1 I think.

          People can plug in their own client. This is configured in the jbossesb-properties.xml. That said I think that by default we should have the best possible client.

          Cheers,

          --Kurt

          • 2. Re: Embedded FTP Server and FTP Clients
            Kurt Stam Master

            A second option is to offer it as a sepate ftp.sar, which we can pre-package with the standalone archive.

            --Kurt

            • 3. Re: Embedded FTP Server and FTP Clients
              Bill Burke Master

              Don't care which release it goes out under, if it is released at all. Just makes testing easier and running the samples easier, IMO.

              As far as pushing and streaming, definately not enough time to do before MP1, or maybe even with the current architecture. They are just ideas.

              • 4. Re: Embedded FTP Server and FTP Clients
                Mark Little Master

                I'd "third" adding it so the integration tests run out of the box. I wouldn't go any further than that until the MP is released. As for improvements for FTP, they're definitely worth adding, but I wonder if we should do this in 5.0 rather than retro-fit it to 4? We can obviously do the retro-fit if there's enough call for it, or one of the community members wants to give it a go.

                • 5. Re: Embedded FTP Server and FTP Clients
                  Bill Burke Master

                  (The following is post MP1 just so that I don't scare the shit outta yas again).

                  BTW, I would look into something similar for Email, for instance Andy Oliver's embeddable Mail Server.

                  I'm also keen on writing JCA adapters for FTP and email. Both inbound and outbound. JCA has a pretty good abstraction and contract for pooling and XA and security integration. Once JBoss 5 comes around we'll be able to embed JCA into Java SE applications and Tomcat and maybe even WLS/Websphere.

                  Another thing, once we have a good deployment, lifecycle, and component model for ESB, it should be fairly easy to write a POJO interface to JCA inbound and outbound adapters. We should be able to even do this with JBoss 4.x JCA and cut and paste relevant parts from MDB code.

                  • 6. Re: Embedded FTP Server and FTP Clients
                    Weston M. Price Master

                    Librados provides this sort of JCA based connectivity (FTP/SMTP). They are a semi-partner of ours

                    IMO any ESB should be nothing more than a core kernel/messaging bus and then a ton of adapters.

                    Think Mach kernel with everything else in user space :-)

                    • 7. Re: Embedded FTP Server and FTP Clients
                      Mark Little Master

                      +1 to Bill.

                      +1 to Weston, though aspects such as fault tolerance, federation, governance etc. have their hooks within the "kernel" too.

                      • 8. Re: Embedded FTP Server and FTP Clients
                        Mark Little Master

                         

                        "weston.price@jboss.com" wrote:
                        Librados provides this sort of JCA based connectivity (FTP/SMTP). They are a semi-partner of ours

                        IMO any ESB should be nothing more than a core kernel/messaging bus and then a ton of adapters.

                        Think Mach kernel with everything else in user space :-)



                        BTW, our ESB is a SOA infrastructure too. That impacts the model because you can have an ESB that fits your description and is entirely based around distributed objects, fine grained method invocations, closely coupled interactions and (implicitly) short duration flows. We definitely don't want to do that, because that's pretty much where ESBs were 3 years ago and things have moved on a lot since then.

                        • 9. Re: Embedded FTP Server and FTP Clients
                          Bill Burke Master

                          Weston, people have to pay for the LIbrados stuff, don't they? It wouldn't be very hard at all to write our own FTP/SMTP/POP/IMap inbound/outbound connectors.

                          Mark, this is what we've been arguing about privately over email. JBPM provides support for long-term flows, and because the flow definition/metadata can either be persisted or travel with the message, federation would be supported.

                          In my mind, even compensating transactions could be covered in the process flow definition. You either rollback in a local transaction to your original save point in the message flow and start over, or you catch the problem and transition to the compensating part of the defined message flow.

                          • 10. Re: Embedded FTP Server and FTP Clients
                            Weston M. Price Master

                             


                            Weston, people have to pay for the LIbrados stuff, don't they? It wouldn't be very hard at all to write our own FTP/SMTP/POP/IMap inbound/outbound connectors.


                            That's why I mentioned that Librados is a semi partner. I believe we have some type of relationship with them that might allow us to deploy their stuff in some version or another. At the very least, I am sure something could be worked out with them for ESB.

                            The question is never about the difficulty, it's about doing the bare amount of research to determine whether or not we can get something like this off the shelf.

                            Let's be realistic what we do is not rocket science, though we may like to think so. Like the ' 1000 monkey's writing Shakespeare' thing, throw enough clowns in front of a terminal, give them enough time and they will eventually write EJB3...oh wait...nevermind :-)

                            IMO we have a hard core case of NIH at JBoss. It permeates most projects and ends up costing us more than we ultimately realize until it's too late (you can insert one of 10 technologies we wrote from scatch in AS in this category).

                            Sure, I could sit down and write my own concurrency abstractions (ThreadPool, Semaphore etc) but why not just use what's in the VM? Similarly, we could sit down and write a bunch of adapters to handle FTP, SMTP, POP. but if we can possibly get a tested piece of code off the shelf, it may be beneficial to look into it.

                            BTW we already have POP/IMAP via the Mail adapter :-)






                            • 11. Re: Embedded FTP Server and FTP Clients
                              Mark Little Master

                              Weston, I think we can all agree that NIH is bad. However, so is believing that everything that has gone before is necessarily the right way of doing things. As far as ESB is concerned (aka SOA infrastructure), we're not in the business of mandating implementation strategies as some of our competitors are. We want to be able to support flexible services for long duration interactions based on SOA and not based on distributed object technologies. That doesn't preclude us supporting the things you've mentioned, but it doesn't prevent users doing it in other ways.

                              • 12. Re: Embedded FTP Server and FTP Clients
                                Mark Little Master

                                 

                                "bill.burke@jboss.com" wrote:
                                Mark, this is what we've been arguing about privately over email. JBPM provides support for long-term flows, and because the flow definition/metadata can either be persisted or travel with the message, federation would be supported.


                                I'm pretty sure we've established that this is a good thing for us to support on the ESB.


                                In my mind, even compensating transactions could be covered in the process flow definition. You either rollback in a local transaction to your original save point in the message flow and start over, or you catch the problem and transition to the compensating part of the defined message flow.


                                Until I see an implementation of a process engine that uses a transaction engine as its core for compensations, I'm not convinced that compensations are truly supported. Something like WS-BA (or one of the other dozen or so compensation transaction models) require durability, coordination and reliability characteristics that are often just ignored by process engines (both commercial and open source), leaving a lot up to the user.

                                • 13. Re: Embedded FTP Server and FTP Clients
                                  Weston M. Price Master

                                   


                                  Weston, I think we can all agree that NIH is bad. However, so is believing that everything that has gone before is necessarily the right way of doing things.


                                  Agreed. My point wasn't to assert that it's necessarily correct, or the right way. It was merely to put into play the idea that these things might be worth evaluating, that's it.

                                  Unfortunately, on the flip side of NIH, we also have the exact opposite issue: incorporating technologies that are not only not a good fit, but don't actually solve the initial problem that they were ostensibly designed to address.






                                  • 14. Re: Embedded FTP Server and FTP Clients
                                    Mark Little Master

                                     

                                    "weston.price@jboss.com" wrote:

                                    Weston, I think we can all agree that NIH is bad. However, so is believing that everything that has gone before is necessarily the right way of doing things.


                                    Agreed. My point wasn't to assert that it's necessarily correct, or the right way. It was merely to put into play the idea that these things might be worth evaluating, that's it.


                                    Absolutely. Even if existing implementations might not look suitable to a "loosely coupled pattern" (just using that as an example), it may be possible to leverage them (e.g., wrap them, or whatever), such that you only end up writing 10% new code. I'd much prefer that to us writing 90% new code.


                                    Unfortunately, on the flip side of NIH, we also have the exact opposite issue: incorporating technologies that are not only not a good fit, but don't actually solve the initial problem that they were ostensibly designed to address.


                                    Yes, I agree with you again. One thing we have to remember with JBossESB is that a lot of the code comes from the Rosetta acquisition and was implemented over 3 years ago. So for example I'm not sure where JBoss JCA was in those days ;-) But purely from a management side of things, if there's something similar/exactly the same elsewhere in JEMS (or other OSS communities), we should look at its applicability. I'd rather you had the headache of support than me!

                                    1 2 Previous Next