csvItemWriter issue

[kk31]

Hi,

         I am trying to develop standalone java batch application which reads data from DB2(v9.7) tables and writes it to csv file with out header with JDK 1.8. 

here is my job.xml

<chunk item-count="10000" >

  <reader ref="jdbcItemReader">

  <properties>

       <property name="sql"   value="xxx />

        <property name="driver-name" value="com.ibm.db2.jcc.DB2Driver" /> 

       <property name="url" value="jdbc:db2://xxx:123/schemaName" />

       <property name="user" value="xxxx" />

       <property name="password" value="xxxx" />

       <property name="beanType" value="java.util.List" />

  </properties>

  </reader>

  <processor ref="xxxxItemProcessor"></processor>

  <writer ref="csvItemWriter">

  <properties>

       <property name="resource" value="test.out" />

       <property name="writeMode" value="overwrite" />

       <property name="preference" value="STANDARD_PREFERENCE" />

       <property name="beanType" value="java.util.List" />

       <property name="header" value="' '" />

  </properties>

  </writer>

  </chunk>

It throws the following error message while writing it into csv by using csvItemWriter.  

Oct 03, 2016 1:25:29 PM org.jberet.runtime.runner.ChunkRunner readProcessWriteItems

ERROR: ProcessingInfo{count=1197, timerExpired=false, itemState=RUNNING, chunkState=DEPLETED, checkpointPosition=-1, readPosition=1197, failurePoint=null}

Oct 03, 2016 1:25:29 PM org.jberet.runtime.runner.ChunkRunner run

ERROR: item-count=10000, time-limit=0, skip-limit=-1, skipCount=0, retry-limit=-1, retryCount=0

Oct 03, 2016 1:25:29 PM org.jberet.runtime.runner.ChunkRunner run

ERROR: JBERET000007: Failed to run job job, xxxxxx, org.jberet.job.model.Chunk@e4fe4dd

java.lang.IllegalArgumentException: columns to write should not be empty on line 4

  at org.supercsv.io.AbstractCsvWriter.writeRow(AbstractCsvWriter.java:162)

  at org.supercsv.io.AbstractCsvWriter.writeRow(AbstractCsvWriter.java:126)

  at org.supercsv.io.CsvListWriter.write(CsvListWriter.java:71)

  at org.jberet.support.io.CsvItemWriter.writeItems(CsvItemWriter.java:152)

  at org.jberet.runtime.runner.ChunkRunner.doCheckpoint(ChunkRunner.java:522)

  at org.jberet.runtime.runner.ChunkRunner.readProcessWriteItems(ChunkRunner.java:317)

  at org.jberet.runtime.runner.ChunkRunner.run(ChunkRunner.java:201)

  at org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:219)

  at org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:140)

  at org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164)

  at org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88)

  at org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:59)

  at org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

  at java.lang.Thread.run(Thread.java:745)

Oct 03, 2016 1:25:29 PM org.jberet.support.io.CsvItemWriter close

INFO: JBERET060502: Closing resource MyBatchletTest_test.out in class org.jberet.support.io.CsvItemWriter

Exception in thread "main" javax.batch.operations.BatchRuntimeException: The job did not complete: job.xml

  at org.jberet.se.Main.main(Main.java:56)

After reading thru other threads     Result set is closed

  I updated my classpath       jberet-support-1.3.0.Beta3.jar file  and still having same issue.

   If we use jacksonCSVItemWriter, it is happy and writes it into file and job is successful .  So is it csvItemWriter issue or jdbcItemReader issue?

Thanks

[cfang]

Can the csvitemwriter write a file without a header?

the above thread discusses a similar issue with csvItemWriter and jacksonCsvItemWriter.

[kk31]

This is total different issue than the which i had posted.   Please take a look at my question once again

[cfang]

is the jdbcItemReader working for you after upgrading to JBeret 1.3.0.Beta3, to address issue

JBERET-253

(jdbcItemReader on db2: Invalid operation: result set is closed. ERRORCODE=-4470, SQLSTATE=null)?

It seems so now that you've got past the reader part and reached writer part.

[kk31]

No its not fixing the issue. That is the reason i asked the question is it jdbcItemReader issue or csvItemWriter issues?

I had latest jar in my classpath,

<classpathentry kind="lib" path="C:/jberet-distribution-1.2.0.Final/lib/jberet-support-1.3.0.Beta3.jar" />

[cfang]

there are 2 issues here:

1, jdbcItemReader (resultSet closed on DB2), I think it should be fixed by upgrading to JBeret 1.3.0.Beta3 to address JBERET-253.  If jdbcItemReader still fails, you would not have reached csvItemWriter part.

2, csvItemWriter (java.lang.IllegalArgumentException: columns to write should not be empty on line 4).  Can you inspect what is on line 4?  Is it an empty line?

[kk31]

// 1, jdbcItemReader (resultSet closed on DB2), I think it should be fixed by upgrading to JBeret 1.3.0.Beta3 to address JBERET-253.  If jdbcItemReader still fails, you would not have reached csvItemWriter part.

             That is my thought earlier also.   But after having a close look at the exception message.          

                        ProcessingInfo{count=1197, timerExpired=false, itemState=RUNNING, chunkState=DEPLETED, checkpointPosition=-1, readPosition=1197, failurePoint=null}

                            After couple of more tests by simply passing jdbcItemReader resultset to csvItemWriter, i was able to generate the file.   So this has been ruled out.

//   2, csvItemWriter (java.lang.IllegalArgumentException: columns to write should not be empty on line 4).  Can you inspect what is on line 4?  Is it an empty line?

  My scenario is, in my itemprocessor file, we have to iterate over all 1197 rows and build query by passing certain columns from 1197 rows and generate the output for csv. 

         The select statement in itemProcessor generates 8k rows and i am able to print them in output log file and i don't see any issue over there. While writing using csvItemWriter, i am getting exception  illegalArgumentException.

            I don't know  what is there on line 4.    In output log, i don't see any empty line.

[cfang]

from looking at super-csv AbstractCsvWriter code:

super-csv/AbstractCsvWriter.java at v2.3.1 · super-csv/super-csv · GitHub

it converts your data item (which is a list) to string array, and if the string array has length 0, it throws this IllegalArgumentException.

Can you run iwth a debugger and see what data is being passed to the writer around that failure point?  The item processor might have generated an empty list, which somehow is not displayed in your log output.

[cfang]

I modified one of my tests, and was able to reproduce your error.  What I did is adding a item processor to job xml that returns a new empty list for one certain item:

//just to reproduce the error...

@Named
@Dependent
public class EmptyLineItemProcessor implements ItemProcessor {

   private int count;

   @Override
   public Object processItem(final Object item) throws Exception {

   if (count == 2 && item instanceof java.util.List) {

   return new ArrayList<String>();
   }

   count++;
  return item;
   }

}

In my output file generated by csvItemWriter, I was able to see all rows before that failing row.  so I suspect something similar is in your item processor.

[cfang]

I think the best approach is to improve the item processor not to produce empty list.  Depending on your requirement in project, maybe just ignore these empty item inside the item processor.

jberet-support csvItemWriter should be more fault-tolerant by checking for these empty item, not to pass them to super-csv writer.

1. JBERET-269

java.lang.IllegalArgumentException: columns to write should not be empty on line 4

[kk31]

Thanks Cheng,   I ran the debugger and fixed my itemProcessor so that it is not going to produce empty list and was able to write it csv file.