10 Replies Latest reply on Oct 5, 2016 10:00 AM by kk31

    csvItemWriter issue

    kk31
    kk31 Oct 3, 2016 2:55 PM

      Hi,

               I am trying to develop standalone java batch application which reads data from DB2(v9.7) tables and writes it to csv file with out header with JDK 1.8. 

       

      here is my job.xml

       

      <chunk item-count="10000" >

        <reader ref="jdbcItemReader">

        <properties>

             <property name="sql"   value="xxx />

              <property name="driver-name" value="com.ibm.db2.jcc.DB2Driver" /> 

             <property name="url" value="jdbc:db2://xxx:123/schemaName" />

             <property name="user" value="xxxx" />

             <property name="password" value="xxxx" />

             <property name="beanType" value="java.util.List" />

        </properties>

        </reader>

        <processor ref="xxxxItemProcessor"></processor>

        <writer ref="csvItemWriter">

        <properties>

             <property name="resource" value="test.out" />

             <property name="writeMode" value="overwrite" />

             <property name="preference" value="STANDARD_PREFERENCE" />

             <property name="beanType" value="java.util.List" />

             <property name="header" value="' '" />

        </properties>

        </writer>

        </chunk>

       

      It throws the following error message while writing it into csv by using csvItemWriter.  

       

      Oct 03, 2016 1:25:29 PM org.jberet.runtime.runner.ChunkRunner readProcessWriteItems

      ERROR: ProcessingInfo{count=1197, timerExpired=false, itemState=RUNNING, chunkState=DEPLETED, checkpointPosition=-1, readPosition=1197, failurePoint=null}

      Oct 03, 2016 1:25:29 PM org.jberet.runtime.runner.ChunkRunner run

      ERROR: item-count=10000, time-limit=0, skip-limit=-1, skipCount=0, retry-limit=-1, retryCount=0

      Oct 03, 2016 1:25:29 PM org.jberet.runtime.runner.ChunkRunner run

      ERROR: JBERET000007: Failed to run job job, xxxxxx, org.jberet.job.model.Chunk@e4fe4dd

      java.lang.IllegalArgumentException: columns to write should not be empty on line 4

        at org.supercsv.io.AbstractCsvWriter.writeRow(AbstractCsvWriter.java:162)

        at org.supercsv.io.AbstractCsvWriter.writeRow(AbstractCsvWriter.java:126)

        at org.supercsv.io.CsvListWriter.write(CsvListWriter.java:71)

        at org.jberet.support.io.CsvItemWriter.writeItems(CsvItemWriter.java:152)

        at org.jberet.runtime.runner.ChunkRunner.doCheckpoint(ChunkRunner.java:522)

        at org.jberet.runtime.runner.ChunkRunner.readProcessWriteItems(ChunkRunner.java:317)

        at org.jberet.runtime.runner.ChunkRunner.run(ChunkRunner.java:201)

        at org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:219)

        at org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:140)

        at org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)

        at org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)

        at org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:59)

        at org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

       

       

      Oct 03, 2016 1:25:29 PM org.jberet.support.io.CsvItemWriter close

      INFO: JBERET060502: Closing resource MyBatchletTest_test.out in class org.jberet.support.io.CsvItemWriter

      Exception in thread "main" javax.batch.operations.BatchRuntimeException: The job did not complete: job.xml

        at org.jberet.se.Main.main(Main.java:56)

       

       

      After reading thru other threads     Result set is closed

        I updated my classpath       jberet-support-1.3.0.Beta3.jar file  and still having same issue.

       

         If we use jacksonCSVItemWriter, it is happy and writes it into file and job is successful .  So is it csvItemWriter issue or jdbcItemReader issue?

       

       

      Thanks

       

       

       

       

       

      • 480 Views
      • Tags:
        • 1. Re: csvItemWriter issue
          cfang
          cfang Oct 3, 2016 3:45 PM (in response to kk31)

          Can the csvitemwriter write a file without a header?

           

          the above thread discusses a similar issue with csvItemWriter and jacksonCsvItemWriter.

            • Actions
          • 2. Re: csvItemWriter issue
            kk31
            kk31 Oct 3, 2016 3:48 PM (in response to cfang)

            This is total different issue than the which i had posted.   Please take a look at my question once again

              • Actions
            • 3. Re: csvItemWriter issue
              cfang
              cfang Oct 3, 2016 3:49 PM (in response to cfang)

              is the jdbcItemReader working for you after upgrading to JBeret 1.3.0.Beta3, to address issue

              JBERET-253

              (jdbcItemReader on db2: Invalid operation: result set is closed. ERRORCODE=-4470, SQLSTATE=null)?

               

              It seems so now that you've got past the reader part and reached writer part.

                • Actions
              • 4. Re: csvItemWriter issue
                kk31
                kk31 Oct 3, 2016 3:54 PM (in response to cfang)

                No its not fixing the issue. That is the reason i asked the question is it jdbcItemReader issue or csvItemWriter issues?

                 

                I had latest jar in my classpath,

                <classpathentry kind="lib" path="C:/jberet-distribution-1.2.0.Final/lib/jberet-support-1.3.0.Beta3.jar" />

                  • Actions
                • 5. Re: csvItemWriter issue
                  cfang
                  cfang Oct 3, 2016 4:13 PM (in response to kk31)

                  there are 2 issues here:

                   

                  1, jdbcItemReader (resultSet closed on DB2), I think it should be fixed by upgrading to JBeret 1.3.0.Beta3 to address JBERET-253.  If jdbcItemReader still fails, you would not have reached csvItemWriter part.

                   

                  2, csvItemWriter (java.lang.IllegalArgumentException: columns to write should not be empty on line 4).  Can you inspect what is on line 4?  Is it an empty line?

                    • Actions
                  • 6. Re: csvItemWriter issue
                    kk31
                    kk31 Oct 4, 2016 10:38 AM (in response to cfang)

                    // 1, jdbcItemReader (resultSet closed on DB2), I think it should be fixed by upgrading to JBeret 1.3.0.Beta3 to address JBERET-253.  If jdbcItemReader still fails, you would not have reached csvItemWriter part.

                                 That is my thought earlier also.   But after having a close look at the exception message.          

                                            ProcessingInfo{count=1197, timerExpired=false, itemState=RUNNING, chunkState=DEPLETED, checkpointPosition=-1, readPosition=1197, failurePoint=null}

                                                After couple of more tests by simply passing jdbcItemReader resultset to csvItemWriter, i was able to generate the file.   So this has been ruled out.

                     

                     

                    //   2, csvItemWriter (java.lang.IllegalArgumentException: columns to write should not be empty on line 4).  Can you inspect what is on line 4?  Is it an empty line?

                                

                      My scenario is, in my itemprocessor file, we have to iterate over all 1197 rows and build query by passing certain columns from 1197 rows and generate the output for csv. 

                             The select statement in itemProcessor generates 8k rows and i am able to print them in output log file and i don't see any issue over there. While writing using csvItemWriter, i am getting exception  illegalArgumentException.

                                I don't know  what is there on line 4.    In output log, i don't see any empty line.

                     

                           

                      • Actions
                    • 7. Re: csvItemWriter issue
                      cfang
                      cfang Oct 4, 2016 11:42 AM (in response to kk31)

                      from looking at super-csv AbstractCsvWriter code:

                      super-csv/AbstractCsvWriter.java at v2.3.1 · super-csv/super-csv · GitHub

                       

                      it converts your data item (which is a list) to string array, and if the string array has length 0, it throws this IllegalArgumentException.

                       

                      Can you run iwth a debugger and see what data is being passed to the writer around that failure point?  The item processor might have generated an empty list, which somehow is not displayed in your log output.

                        • Actions
                      • 8. Re: csvItemWriter issue
                        cfang
                        cfang Oct 4, 2016 5:32 PM (in response to cfang)

                        I modified one of my tests, and was able to reproduce your error.  What I did is adding a item processor to job xml that returns a new empty list for one certain item:

                         

                        //just to reproduce the error...

                        @Named
                        @Dependent
                        public class EmptyLineItemProcessor implements ItemProcessor {

                           private int count;

                           @Override
                           public Object processItem(final Object item) throws Exception {

                           if (count == 2 && item instanceof java.util.List) {

                           return new ArrayList<String>();
                           }

                           count++;
                          return item;
                           }

                        }

                         

                        In my output file generated by csvItemWriter, I was able to see all rows before that failing row.  so I suspect something similar is in your item processor.

                          • Actions
                        • 9. Re: csvItemWriter issue
                          cfang
                          cfang Oct 4, 2016 5:50 PM (in response to cfang)

                          I think the best approach is to improve the item processor not to produce empty list.  Depending on your requirement in project, maybe just ignore these empty item inside the item processor.

                           

                          jberet-support csvItemWriter should be more fault-tolerant by checking for these empty item, not to pass them to super-csv writer.

                           

                          1. JBERET-269

                          java.lang.IllegalArgumentException: columns to write should not be empty on line 4

                            • Actions
                          • 10. Re: csvItemWriter issue
                            kk31
                            kk31 Oct 5, 2016 10:00 AM (in response to cfang)

                            Thanks Cheng,   I ran the debugger and fixed my itemProcessor so that it is not going to produce empty list and was able to write it csv file.

                              • Actions