9 Replies Latest reply on Oct 5, 2016 11:05 AM by richardmoore

    Fixed formatted records

    richardmoore

      I have tried to duplicate the BeanIO example for the star-entities and am getting the following -

       

      2016-10-04 15:38:01.006 [jberet-1    ]                         jberet - [ERROR] ProcessingInfo{count=0, timerExpired=false, itemState=RUNNING, chunkState=RUNNING, checkpointPosition=-1, readPosition=0, failurePoint=null}

      2016-10-04 15:38:01.037 [jberet-1    ]                         jberet - [ERROR] item-count=10, time-limit=0, skip-limit=-1, skipCount=0, retry-limit=-1, retryCount=0

      2016-10-04 15:38:01.065 [jberet-1    ]                         jberet - [ERROR] JBERET000007: Failed to run job Jberet_Adhoc_TestCase3, step1, org.jberet.job.model.Chunk@f905252

      java.lang.NullPointerException

        at org.jberet.support.io.ItemReaderWriterBase.validate(ItemReaderWriterBase.java:92) ~[jberet-support-1.3.0.Beta3.jar:?]

        at org.jberet.support.io.BeanIOItemReader.readItem(BeanIOItemReader.java:119) ~[jberet-support-1.3.0.Beta3.jar:?]

        at org.jberet.runtime.runner.ChunkRunner.readItem(ChunkRunner.java:359) ~[jberet-core-1.3.0.Beta3.jar:1.3.0.Beta3]

        at org.jberet.runtime.runner.ChunkRunner.readProcessWriteItems(ChunkRunner.java:305) ~[jberet-core-1.3.0.Beta3.jar:1.3.0.Beta3]

        at org.jberet.runtime.runner.ChunkRunner.run(ChunkRunner.java:201) [jberet-core-1.3.0.Beta3.jar:1.3.0.Beta3]

        at org.jberet.runtime.runner.StepExecutionRunner.runBatchletOrChunk(StepExecutionRunner.java:226) [jberet-core-1.3.0.Beta3.jar:1.3.0.Beta3]

        at org.jberet.runtime.runner.StepExecutionRunner.run(StepExecutionRunner.java:147) [jberet-core-1.3.0.Beta3.jar:1.3.0.Beta3]

        at org.jberet.runtime.runner.CompositeExecutionRunner.runStep(CompositeExecutionRunner.java:164) [jberet-core-1.3.0.Beta3.jar:1.3.0.Beta3]

        at org.jberet.runtime.runner.CompositeExecutionRunner.runFromHeadOrRestartPoint(CompositeExecutionRunner.java:88) [jberet-core-1.3.0.Beta3.jar:1.3.0.Beta3]

        at org.jberet.runtime.runner.JobExecutionRunner.run(JobExecutionRunner.java:60) [jberet-core-1.3.0.Beta3.jar:1.3.0.Beta3]

        at org.jberet.spi.JobExecutor$1.run(JobExecutor.java:99) [jberet-core-1.3.0.Beta3.jar:1.3.0.Beta3]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_73]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_73]

        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_73]

       

      I downloaded the ca2013entities_ascii.txt file and the org.jberet.support.io.CaliforniaStarEntity source from gitHub, and am using the step -

       

      <chunk>

        <reader ref="beanIOItemReader">

                      <properties>

                          <property name="resource" value="#{systemProperties['APP_DATA_PERSISTENT_FCLTY']}/#{systemProperties['batchJobname']}.txt"/>

                          <property name="streamName" value="star-entities"/>

                          <property name="streamMapping" value="#{systemProperties['APP_PARMS_FCLTY']}/#{systemProperties['batchJobname']}_Bean.xml"/>

                          <property name="mappingProperties" value="zipCodeFieldName=zipCode, zipCodeFieldType=string"/>

                          <property name="errorHandler" value="org.beanio.BeanReaderErrorHandlerSupport"/>

                          <property name="charset" value="UTF-8"/>

                      </properties>

                  </reader>

                  <writer ref="beanIOItemWriter">

                      <properties>

                          <property name="resource" value="#{systemProperties['APP_DATA_TMP_FCLTY']}/#{systemProperties['batchJobname']}.out"/>

                          <property name="streamName" value="star-entities"/>

                          <property name="streamMapping" value="#{systemProperties['APP_PARMS_FCLTY']}/#{systemProperties['batchJobname']}_Bean.xml"/>

                          <property name="mappingProperties" value="zipCodeFieldName=zipCode, zipCodeFieldType=string"/>

                          <property name="charset" value="UTF-8"/>

                      </properties>

                  </writer>

        </chunk>

       

      Got any ideas what I have wrong? Also, is this the best way to process fixed format records?

        • 1. Re: Fixed formatted records
          cfang

          BeanIO is the underlying library jberet-support uses to support fixed length format.  I'll take a look.

          • 2. Re: Fixed formatted records
            cfang

            the NPE is from invoking bean validation.  It could be that the validator is null.  Can you disable bean validation by setting itrem reader property

             

            <property name="skipBeanValidation" value="true"/>

             

            The validator is null probably because hibernate-validator (the bean-validation spec implementation) is not in your  classpath.

            • 3. Re: Fixed formatted records
              cfang

              the same NPE-from-bean-validator issue in Re: jdbcItemReader with POJO beanType case problem

              • 4. Re: Fixed formatted records
                cfang

                Does this NullPointerException occur every time repeatably, or intermitten?

                • 5. Re: Fixed formatted records
                  richardmoore

                  When I added -

                              <reader ref="beanIOItemReader">

                                  <properties>

                                      <property name="resource" value="#{systemProperties['APP_DATA_PERSISTENT_FCLTY']}/#{systemProperties['batchJobname']}.txt"/>

                                      <property name="streamName" value="star-entities"/>

                                      <property name="streamMapping" value="#{systemProperties['APP_PARMS_FCLTY']}/#{systemProperties['batchJobname']}_Bean.xml"/>

                                      <property name="mappingProperties" value="zipCodeFieldName=zipCode, zipCodeFieldType=string"/>

                                      <property name="errorHandler" value="org.beanio.BeanReaderErrorHandlerSupport"/>

                                      <property name="skipBeanValidation" value="true"/>

                                      <property name="charset" value="UTF-8"/>

                                  </properties>

                              </reader>

                  It hangs and never completes the step.

                  When I added in the hibernate validator jars (including those they identify as required) -

                  It hangs and never completes the step.

                  When I removed the skipBeanValidation tag, even when the hibernate validator jars are included on the classpath the NPE on the validate method occurs.

                  To your other question, it happens every time.

                  • 6. Re: Fixed formatted records
                    richardmoore

                    I ended up letting it process and it rand for :12 to process the 11,472 records.

                    • 7. Re: Fixed formatted records
                      richardmoore

                      When it completes I do not see the message of the resource and filename being closed on the output side like I do with the jackson csv writer. Should I be seeing it?

                      • 8. Re: Fixed formatted records
                        cfang

                        beanIOItemWriter close() method does not log that message, which could be an oversight.

                         

                        :12 to process the 11,472 records, that seems pretty long processing time.  Did you allocate enough memory for the jvm?

                        • 9. Re: Fixed formatted records
                          richardmoore

                          I am running on my virtual desktop and yes it is slow. I added the item-count="3000" attribute and it ripped through them in 12 seconds. We are sitting good. Thanks so much for your help. Sorry I didn't catch that this was essentially the same problem I had with the jdbc reader with bean processing.