0 Replies Latest reply on Feb 15, 2012 8:34 AM by vs.souza

    Performance issues to load csv file

    vs.souza

      Hello fellows,

       

      I created a bundle to load a CSV file, validate some information and send it to an ActiveMQ queue. The problem is that my CSV file has 30.000.000 registers and about 1.2 GB and my bundle is taking about 12 hours to load 750.000 itens to the queue. My ServiceMix is 4.4.3 and I am loading the file woking wit camel and bindy. Bellow I post my camel-context.xml and my bindy class:

       

      camel-context.xml:

       

       

                <camel:log message="Started unmarshalling file $ at $..."/>

            <camel:split streaming="true">

            <camel:tokenize token="\n"/>

            <camel:unmarshal ref="bindyDataFormat"/>

            <camel:to uri="activemq:filemove-events"/>

            </camel:split>

            <camel:log message="Finished unmarshalling file $ at $..."/>

          </camel:route>

           

           

        </camel:camelContext>

       

      </beans>

       

      bindy bean:

       

      package com.test.integration.camel.spring.poc.file;

       

      import java.io.Serializable;

       

      import org.apache.camel.dataformat.bindy.annotation.CsvRecord;

      import org.apache.camel.dataformat.bindy.annotation.DataField;

       

      @CsvRecord(separator=",", quote="\"")

      public class CSVEventRecordBean implements Serializable{

       

              private static final long serialVersionUID = -8806841912643394977L;

       

              @DataField(pos=1)

              private String eventDate;

               

              @DataField(pos=2)

              private String userId;

               

              @DataField(pos=3)

              private String systemId;

       

              public String getEventDate() {

                      return eventDate;

              }

       

              public void setEventDate(String eventDate) {

                      this.eventDate = eventDate;

              }

       

              public String getUserId() {

                      return userId;

              }

       

              public void setUserId(String userId) {

                      this.userId = userId;

              }

       

              public String getSystemId() {

                      return "Bean Generated: " + systemId;

              }

       

              public void setSystemId(String systemId) {

                      this.systemId = systemId;

              }

               

      }

       

       

      How can I improve the performance considerably to make it faster? Do you have any suggestions?

       

      Cheers.

       

      Edited by: vs.souza on Feb 15, 2012 1:34 PM