Batch Processing And Transaction Rollback

[nathandennis]

I am trying to preform batch processing on a large amount of data. Start from a quartz cron job, I'm collecting records from the DB , moving these records to an arraylist, and remotely connecting to a foreign server to processing foreign transaction for each record retrieved. my question is, if something go a miss and the seams managed transaction breaks due to an exception,

how can i keep the entire group of changes from being rolled back?  some code....

@Transactional
@Name("startupBilling")
public class StartupBilling {

     @Logger
     private Log log;     
     
     @In
     AsyncBillingProcessor asyncBillingProcessor;
     @Observer("org.jboss.seam.postInitialization")
     public void observe() 
     {
          try
          {
               log.info("Trying to Start Cron Job");
               Calendar cal = Calendar.getInstance ();
               cal.set (2040, Calendar.MAY, 10);
               QuartzTriggerHandle handle = asyncBillingProcessor.scheduleBillingQue(new Date(), "0 0 0 * * ?", cal.getTime());
               
          }
          catch(Exception e)
          {
               e.printStackTrace();
               log.info("Operation Failed");
          }
          log.info("Job Started");

     }

}

@Name("asyncBillingProcessor")  
@AutoCreate
public class AsyncBillingProcessor {

     
     @Logger Log log;
     
     @In 
     EntityManager entityManager;
     
     @SuppressWarnings("unchecked")
     @Asynchronous
     @Transactional @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
     public QuartzTriggerHandle scheduleBillingQue(@Expiration Date when,                                                             @IntervalCron String cron,
                                                             @FinalExpiration Date endDate)
     {
          ArrayList<Systemsettings> ss = (ArrayList<Systemsettings>) entityManager.createQuery("from Systemsetting where name=:param")
                    .setParameter("param", "recurringbilling")
                    .getSingleResult();
          
          if(ss.size() > 0){
               if(ss.get(0).isSw1()){
                    RecurringBillingLocal rb = (RecurringBillingLocal) Component.getInstance("recurringBillingAction");
                    Boolean  sw = rb.billingProcess();
                    log.info("billing processed successfully? " + sw);
               } 
          }
          return null;
     }     
}

@Stateful
@Name("recurringBillingAction")
@Scope(ScopeType.EVENT)
public class RecurringBillingAction implements RecurringBillingLocal {
     


     @PersistenceContext
     private EntityManager entityManager;

     
     
     @Logger
     private Log log;
     @In
     private Renderer renderer;
     
     private ArrayList<Userlogin> userlogin;
     
     private ArrayList<UserloginInvoiceWrapper> uli = new ArrayList<UserloginInvoiceWrapper>();
     private UserloginInvoiceWrapper activeuser;
     private Userlogin activeadmin;
     

     public Boolean billingProcess(){
          Calendar today = Calendar.getInstance();
          Calendar leap = Calendar.getInstance();
          leap.set(today.get(Calendar.YEAR), Calendar.FEBRUARY, 28);
          
          try{
               if(today.get(Calendar.MONTH)==0){//Calendar.Month starts at 0 = jan
                    log.info("we are in january");
                    this.userlogin = (ArrayList<Userlogin>) 
                         entityManager.createQuery("from Userlogin where year=:lastyear and month=9 and day=:day")
                                                         .setParameter("lastyear", today.get(Calendar.YEAR) -1)
                                                         .setParameter("day", today.get(Calendar.DAY_OF_MONTH))
                                                         .getResultList();
.....

based on date grab the matching records and preform the billing.

how can i stop a transaction rollback on the entire record set in the event on record fails?
i realize this might require me to change the design slightly.

i've seen annotation such as

@ApplicationException(rollback=false)

that i have a feeling might be helpful, but im not sure. any direction would be greatly appreciated.

[kapitanpetko]

You if have the option of using EJB's, it might be as easy as using a separate REQUIRES_NEW method for each entry in your list.
If not, well, Seam cannot help you, I think, so you might need to do your own transaction management (UserTransaction).

[jeanluc]

Some options: don't attempt to do everything in a single transaction. Select a bunch of records, process them, commit, then repeat. If you don't have a requirement to process all or none (which you don't appear to be the case), there is no reason to attempt to do everything in one transaction.

You may also want to look at using savepoints, but it may be too much.

[hcgpragt]

Would there be a difference in using
@in PersistenceContext billingDataSource;
instead of

@EntityManager
PersistenceContext em;

??

[jeanluc]

There is a technical difference (I presume you actually meant @PersistenceContext EntityManager em) and the difference is Seam- vs container-managed persistence context, but as a quick thought I don't see how this would be a relevant different for your case, which is about batching changes, not about who manages the PC

[nathandennis]

thanks for the responses. just what i needed to jar a thought loose in there. just so other can follow in the future and possibly to get any criticism.

heres what i did,

changed the layout of the RecurringBillingAction a little. retrieved my list just as above, but instead of processing in the same bean, moved that logic to a new bean. called via Component.getInstance
and annotated as follows.

     @In
     private EntityManager entityManager;
     
     @Logger
     private Log log;
     @In
     private Renderer renderer;

     private UserloginInvoiceWrapper activeuser;
     private Userlogin activeadmin;
     
     @Transactional @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
     public Boolean process(Userlogin userlogin){

        //anymethod call in here
        }

     @TransactionAttribute(TransactionAttributeType.SUPPORTS)
     public Boolean resign(UserloginInvoiceWrapper uli){

seems to work just fine. intentionally breaking it throws the exception as expected, but since this is moved into its own transaction previous processed records are not rolled back. of course i am flushing the entityManager after each transaction as well.

one last point that we should probably touch on. since hibernate is caching these will the flush release it, or do i need to also call an

entityManager.clear()

aswell to avoid memory problems with very large batchs? thanks for the help guys.

[kapitanpetko]

A few notes about your code:

• since this works from a Quartz job, there is no gain in using SMPC. @PersistenceContext is sufficient.


• you shouldn't mix CMT and Seam managed transactions (no need for @Transactional). It will probably work as is, but it is confusing.


• if you use @PersistenceContext, the EM will be automatically flushed when the transaction commits, so no need to call flush.

As for caching, flush doesn't clear the EM. But if you call clear, that will clear everything, so subsequent calls
might fail. If you really have a lot of records and are worried about memory use, you could retrieve, say, 100 records,
process, clear the EM, repeat. If you are using Hibernate, you can use cursors, so no need to load everything at once.