4 Replies Latest reply on Oct 27, 2015 6:46 AM by _amnesiac

    StaleObjectException at the end of the process triggered from timer

    _amnesiac
    _amnesiac Jul 14, 2015 5:47 AM

      Hi,

       

      I have issue with process started with timer.

       

      We have 2 instances of JBPM 6.1.Final, synchronized with quartz and process definition triggered with timer.

      I see in logs that all bussiness logic (read from DB, sending request via REST) happened, but exception at the very end causes quartz to retry - so it starts again and again until it reach max retry count or until it successfully commit transaction.

      This problem is probably random - today we've noticed it for the first time and after third wrong try it finally have commited.

       

      I'll appreciate any help.

       

      Below pasted logs:

      08:00:09.867 [jBPMClusteredScheduler_Worker-5] WARN  org.drools.persistence.jta.JtaTransactionManager - Unable to commit transaction

      bitronix.tm.internal.BitronixRollbackException: RuntimeException thrown during beforeCompletion cycle caused transaction rollback

        at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:241) ~[btm-2.1.4.jar:2.1.4]

        at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:143) ~[btm-2.1.4.jar:2.1.4]

        at org.drools.persistence.jta.JtaTransactionManager.commit(JtaTransactionManager.java:209) ~[drools-persistence-jpa-6.1.0.Final.jar:6.1.0.Final]

        at org.drools.persistence.SingleSessionCommandService$TransactionInterceptor.execute(SingleSessionCommandService.java:512) [drools-persistence-jpa-6.1.0.Final.jar:6.1.0.Final]

        at org.drools.persistence.SingleSessionCommandService.execute(SingleSessionCommandService.java:353) [drools-persistence-jpa-6.1.0.Final.jar:6.1.0.Final]

        at org.jbpm.process.core.timer.impl.GlobalTimerService$DisposableCommandService.execute(GlobalTimerService.java:281) [jbpm-flow-6.1.0.Final.jar:6.1.0.Final]

        at org.jbpm.persistence.timer.GlobalJpaTimerJobInstance.call(GlobalJpaTimerJobInstance.java:68) [jbpm-persistence-jpa-6.1.0.Final.jar:6.1.0.Final]

        at org.jbpm.persistence.timer.GlobalJpaTimerJobInstance.call(GlobalJpaTimerJobInstance.java:46) [jbpm-persistence-jpa-6.1.0.Final.jar:6.1.0.Final]

        at org.jbpm.process.core.timer.impl.QuartzSchedulerService$QuartzJob.execute(QuartzSchedulerService.java:279) [jbpm-flow-6.1.0.Final.jar:6.1.0.Final]

        at org.quartz.core.JobRunShell.run(JobRunShell.java:216) [quartz-1.8.5.jar:na]

        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549) [quartz-1.8.5.jar:na]

      Caused by: javax.persistence.OptimisticLockException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [org.drools.persistence.info.SessionInfo#62]

        at org.hibernate.ejb.AbstractEntityManagerImpl.wrapStaleStateException(AbstractEntityManagerImpl.java:1416) ~[hibernate-entitymanager-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1329) ~[hibernate-entitymanager-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310) ~[hibernate-entitymanager-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1316) ~[hibernate-entitymanager-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.ejb.AbstractEntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(AbstractEntityManagerImpl.java:1510) ~[hibernate-entitymanager-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:110) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:53) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at bitronix.tm.BitronixTransaction.fireBeforeCompletionEvent(BitronixTransaction.java:532) ~[btm-2.1.4.jar:2.1.4]

        at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:235) ~[btm-2.1.4.jar:2.1.4]

        ... 10 common frames omitted

      Caused by: org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [org.drools.persistence.info.SessionInfo#62]

        at org.hibernate.persister.entity.AbstractEntityPersister.check(AbstractEntityPersister.java:2521) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3240) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:3138) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3468) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:140) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:395) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:387) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:304) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:349) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1159) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:404) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:105) ~[hibernate-core-4.2.12.Final.jar:4.2.12.Final]

        ... 13 common frames omitted

      08:00:09.867 [jBPMClusteredScheduler_Worker-5] WARN  org.drools.persistence.SingleSessionCommandService - Could not commit session

      java.lang.RuntimeException: Unable to commit transaction

        at org.drools.persistence.jta.JtaTransactionManager.commit(JtaTransactionManager.java:212) ~[drools-persistence-jpa-6.1.0.Final.jar:6.1.0.Final]

        at org.drools.persistence.SingleSessionCommandService$TransactionInterceptor.execute(SingleSessionCommandService.java:512) ~[drools-persistence-jpa-6.1.0.Final.jar:6.1.0.Final]

        at org.drools.persistence.SingleSessionCommandService.execute(SingleSessionCommandService.java:353) [drools-persistence-jpa-6.1.0.Final.jar:6.1.0.Final]

        at org.jbpm.process.core.timer.impl.GlobalTimerService$DisposableCommandService.execute(GlobalTimerService.java:281) [jbpm-flow-6.1.0.Final.jar:6.1.0.Final]

        at org.jbpm.persistence.timer.GlobalJpaTimerJobInstance.call(GlobalJpaTimerJobInstance.java:68) [jbpm-persistence-jpa-6.1.0.Final.jar:6.1.0.Final]

        at org.jbpm.persistence.timer.GlobalJpaTimerJobInstance.call(GlobalJpaTimerJobInstance.java:46) [jbpm-persistence-jpa-6.1.0.Final.jar:6.1.0.Final]

        at org.jbpm.process.core.timer.impl.QuartzSchedulerService$QuartzJob.execute(QuartzSchedulerService.java:279) [jbpm-flow-6.1.0.Final.jar:6.1.0.Final]

        at org.quartz.core.JobRunShell.run(JobRunShell.java:216) [quartz-1.8.5.jar:na]

        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549) [quartz-1.8.5.jar:na]

        • 1. Re: StaleObjectException at the end of the process triggered from timer
          swiderski.maciej
          swiderski.maciej Jul 14, 2015 1:40 PM (in response to _amnesiac)

          some questions:

          - what strategy do you use for runtime manager?

          - if it is singleton, make sure the do not share same session id - which in case of singleton is stored in serialized file inside JBOSS_HOME/standalone/data

          - have you configured quartz to be clustered- to not fire jobs on all instances?

           

          if you could share you configuration of quartz that might be useful.

           

          HTH

            • Actions
          • 2. Re: StaleObjectException at the end of the process triggered from timer
            _amnesiac
            _amnesiac Jul 15, 2015 9:09 AM (in response to swiderski.maciej)

            - Chosen runtime strategy is singleton

             

             

            - I don't know how to check this - we use tomcat. Is this feature (session sharing) is configurable or it is resolved on the code level?

             

            In one of the org.kie.internal.executor.api.Command implementation in other process definition (not in timer triggered) we have code like this:

             

             

              final WorkItem workItem = (WorkItem) commandContext.getData("workItem");

                    RuntimeManager runtimeManager = RuntimeManagerRegistry.get().getManager(deploymentId);

              RuntimeEngine engine = runtimeManager.getRuntimeEngine(ProcessInstanceIdContext.get((Long) commandContext.getData("processInstanceId")))

              KieSession kieSession = engine.getKieSession();

              execute(workItem, kieSession);

             

            Do you think this could be wrong approach when singleton strategy is chosen?

             

             

            - processes executes only on one instance, here is the quartz config:

            #============================================================================

            # Configure Main Scheduler Properties 

            #============================================================================

            org.quartz.scheduler.instanceName = jBPMClusteredScheduler

            org.quartz.scheduler.instanceId = AUTO

            #============================================================================

            # Configure ThreadPool 

            #============================================================================

            org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool

            org.quartz.threadPool.threadCount = 5

            org.quartz.threadPool.threadPriority = 5

            #============================================================================

            # Configure JobStore 

            #============================================================================

            org.quartz.jobStore.misfireThreshold = 60000

            org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreCMT

            org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.StdJDBCDelegate

            org.quartz.jobStore.useProperties=false

            org.quartz.jobStore.dataSource=managedDS

            org.quartz.jobStore.nonManagedTXDataSource=notManagedDS

            org.quartz.jobStore.tablePrefix=QRTZ_

            org.quartz.jobStore.isClustered=true

            org.quartz.jobStore.clusterCheckinInterval = 20000

            #============================================================================

            # Configure Datasources 

            #============================================================================

            org.quartz.dataSource.notManagedDS.jndiURL=java:comp/env/jdbc/notManagedQuartzDS

            org.quartz.dataSource.managedDS.jndiURL=java:comp/env/jdbc/jbpm

              • Actions
            • 3. Re: StaleObjectException at the end of the process triggered from timer
              swiderski.maciej
              swiderski.maciej Jul 15, 2015 11:59 AM (in response to _amnesiac)

              Lukasz Dworski wrote:

               

              - Chosen runtime strategy is singleton

               

               

              - I don't know how to check this - we use tomcat. Is this feature (session sharing) is configurable or it is resolved on the code level?

              I believe this is the issue, assuming both tomcat instances are running on same host. Since this is not jboss it will not use the location that I mentioned and instead will use java.io.tmpdir which will be the same I believe as both tomcat instances run via same java. You can override that by setting different directories via system property jbpm.data.dir

               

              so make sure that each tomcat instance that you start have following system property set:

              -Djbpm.data.dir=/my/data/tomcat1

              -Djbpm.data.dir=/my/data/tomcat2

               

              with that it should not use same ksession on both instances.

               

              HTH

                • Actions
              • 4. Re: StaleObjectException at the end of the process triggered from timer
                _amnesiac
                _amnesiac Oct 27, 2015 6:46 AM (in response to swiderski.maciej)

                Our instances are on the separate machines, so this is not the case.

                What else could be the reason for this kind of exception?

                 

                TIA

                  • Actions