7 Replies Latest reply on Mar 31, 2008 4:58 AM by kconner

    Handling Large Files in JBossESB FileGateway

    mussab.sharif

      Dear All,

      I need to make JbossESB work as a file gwatey for files supplied by other system.

      The files normally are large in size (10MB in average) and are in a binary format (ASN.1 Basic Encoding Rules (BER)).

      I have built the decoder In a previous application using a multi threaded stand alone application, and I need to use the same logic to process the files and then store it in a database after its decoded.

      My previous application reads the files using the BufferedInputStream to read the large file in a small pieces, and process the file and then continue reading by using (byte)input.read() method.

      Just note I need to read the file byte by byte to do the appropriate decoding.

      My question is how can configure the ESB to handle the large files in and make it read byte by bye and then putting the output in some kind of database without having a memory problem (which I had when I used the defulet file composer which take the file contents and put it in a message object)?

      what that be solved by building a custom file composer? or there are other ways?


      Thanks

        • 1. Re: Handling Large Files in JBossESB FileGateway
          tfennelly

          You should be able to do this by writing a custom MessageComposer (org.jboss.soa.esb.listeners.message.MessageComposer). e.g. a "FileInputStreamMessageComposer" that simply opens the file InputStream on the compose() call, attaches the stream as the message payload for processing by the action pipeline, and closes the InputStream on the decompose() call.

          We'd be happy to receive this back as a contribution :-)

          • 2. Re: Handling Large Files in JBossESB FileGateway
            tfennelly

            Forgot to mention... you configure your new composer impl on the FileGateway via the "composer-class" property.

            • 3. Re: Handling Large Files in JBossESB FileGateway
              mussab.sharif

              Thanks,

              I will try to do and share the result.

              Thank you again

              • 4. Re: Handling Large Files in JBossESB FileGateway
                jmorgan

                I tried doing this exact thing but ran into serialization/marshalling issues with the FileInputStream. I ended up writing the message composer to just pass the path to the file and let the consumers open it.

                Any suggestions on how to marshall the FileInputStream or how that'd work?

                • 5. Re: Handling Large Files in JBossESB FileGateway
                  tfennelly

                  Not sure which version of the ESB you're using, but Daniel Bevenius has just added a new quickstart to the SVN trunk that demonstrates how you could do this. I think he had to do something similar to you in that he had to pass the file name instead of a stream (which was a bit of a pain - will be sorted in v4.3 I hope). This quickstart is still a work in progress I think (correct me if I'm wrong Dan). It uses Smooks v1.0, which has added SAX support as well as some new routing features and more...

                  Please try this stuff out and let us know what you think!

                  • 6. Re: Handling Large Files in JBossESB FileGateway
                    tfennelly

                    BTW... I think Daniel has successfully tested this Quickstart using a 2Gb+ file as the input message.

                    • 7. Re: Handling Large Files in JBossESB FileGateway
                      kconner

                       

                      "jmorgan" wrote:
                      Any suggestions on how to marshall the FileInputStream or how that'd work?

                      This is not possible in the current architecture as FileInputStreams are not serializable and will therefore fail as soon as the message needs to be marshalled.

                      At present the only solutions are to pass the contents through an out of band mechanism (such as including the filename or storing in a database) or to split up the content into small messages at the gateway.

                      Transparent handling of streams is on the roadmap for a future release.