-
1. Re: Processes getting stuck randomly for no apparent reason
swiderski.maciej Aug 28, 2014 2:03 AM (in response to zpirate)this indicates that event issued by task service on task completion did not reach ksession to move process instance forward. There might be several reasons for it:
- at the time when task was completed there was no RuntimeManager for given deployment
- transaction related issue that caused incorrect check of task state
- no process instance listening for that event
All of these are rarely to happen but might happen in very specific situations. Any way all of them should result in some errors in logs. Have you found anything there?
HTH
-
2. Re: Processes getting stuck randomly for no apparent reason
zpirate Sep 2, 2014 7:30 AM (in response to swiderski.maciej)Hi Maciej, thanks for your reply. You are right, there are some exceptions in the logs that we have ignored so far.
We have analyzed the server logs for two occasions when the issue happened (different dates, different process definitions) and in both cases there is a "Could not commit session..." exception in org.jbpm.services.task.persistence.TaskTransactionInterceptor during the starting of the task that later gets stuck. Although the the root cause of the exception appears to be different, the exception and the behavior afterwards is the same.
Does this help?
-----------------------------------------------------------
Exception 1:
11:47:44,780 WARN [org.jbpm.services.task.persistence.TaskTransactionInterceptor] (http--0.0.0.0-8080-2) Could not commit session: java.lang.IllegalArgumentException: null source
at java.util.EventObject.<init>(EventObject.java:38) [rt.jar:1.6.0_34]
at org.jbpm.services.task.events.TaskEventImpl.<init>(TaskEventImpl.java:17) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.events.TaskEventSupport.fireBeforeTaskStarted(TaskEventSupport.java:43) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.internals.lifecycle.MVELLifeCycleManager.taskOperation(MVELLifeCycleManager.java:304) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.identity.UserGroupLifeCycleManagerDecorator.taskOperation(UserGroupLifeCycleManagerDecorator.java:46) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.impl.TaskInstanceServiceImpl.start(TaskInstanceServiceImpl.java:200) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.commands.StartTaskCommand.execute(StartTaskCommand.java:48) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.commands.StartTaskCommand.execute(StartTaskCommand.java:30) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.commands.CompositeCommand.execute(CompositeCommand.java:38) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.commands.TaskCommandExecutorImpl$SelfExecutionCommandService.execute(TaskCommandExecutorImpl.java:65) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.drools.core.command.impl.AbstractInterceptor.executeNext(AbstractInterceptor.java:41) [drools-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.persistence.TaskTransactionInterceptor.execute(TaskTransactionInterceptor.java:53) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.commands.TaskCommandExecutorImpl.execute(TaskCommandExecutorImpl.java:40) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.impl.command.CommandBasedTaskService.start(CommandBasedTaskService.java:245) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.console.ng.ht.backend.server.TaskServiceEntryPointImpl.startBatch(TaskServiceEntryPointImpl.java:306) [jbpm-console-ng-human-tasks-backend-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.console.ng.ht.backend.server.TaskServiceEntryPointImpl$Proxy$_$$_WeldClientProxy.startBatch(TaskServiceEntryPointImpl$Proxy$_$$_WeldClientProxy.java) [jbpm-console-ng-human-tasks-backend-6.0.1.Final.jar:6.0.1.Final]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_34]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_34]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_34]
at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_34]
at org.jboss.errai.bus.server.io.AbstractRPCMethodCallback.invokeMethodFromMessage(AbstractRPCMethodCallback.java:48) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.io.VoidRPCEndpointCallback.callback(VoidRPCEndpointCallback.java:20) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.io.RemoteServiceCallback.callback(RemoteServiceCallback.java:54) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.cdi.server.CDIExtensionPoints$3.callback(CDIExtensionPoints.java:499) [errai-weld-integration-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.DeliveryPlan.deliver(DeliveryPlan.java:47) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.ServerMessageBusImpl.sendGlobal(ServerMessageBusImpl.java:284) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.SimpleDispatcher.dispatchGlobal(SimpleDispatcher.java:46) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.service.ErraiServiceImpl.store(ErraiServiceImpl.java:92) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.service.ErraiServiceImpl.store(ErraiServiceImpl.java:109) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.servlet.DefaultBlockingServlet.doPost(DefaultBlockingServlet.java:140) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jboss.weld.servlet.ConversationPropagationFilter.doFilter(ConversationPropagationFilter.java:62) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jboss.solder.servlet.exception.CatchExceptionFilter.doFilter(CatchExceptionFilter.java:65) [solder-impl-3.2.1.Final.jar:3.2.1.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jboss.solder.servlet.event.ServletEventBridgeFilter.doFilter(ServletEventBridgeFilter.java:74) [solder-impl-3.2.1.Final.jar:3.2.1.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jbpm.designer.web.filter.impl.PluggableFilter.doFilter(PluggableFilter.java:70) [jbpm-designer-backend-6.0.1.Final.jar:6.0.1.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.uberfire.security.server.UberFireSecurityFilter.doFilter(UberFireSecurityFilter.java:266) [uberfire-security-server-0.3.1.Final.jar:0.3.1.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:]
at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:416) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_34]
-----------------------------------------------------------
Exception 2:
11:42:36,814 WARN [org.jbpm.services.task.persistence.TaskTransactionInterceptor] (http--0.0.0.0-8080-3) Could not commit session: org.jbpm.services.task.exception.PermissionDeniedException: User '[UserImpl:'erik']' was unable to execution operation 'Start' on task id 161 due to a no 'current status' match
at org.jbpm.services.task.internals.lifecycle.MVELLifeCycleManager.evalCommand(MVELLifeCycleManager.java:128) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.internals.lifecycle.MVELLifeCycleManager.taskOperation(MVELLifeCycleManager.java:318) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.identity.UserGroupLifeCycleManagerDecorator.taskOperation(UserGroupLifeCycleManagerDecorator.java:46) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.impl.TaskInstanceServiceImpl.start(TaskInstanceServiceImpl.java:200) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.commands.StartTaskCommand.execute(StartTaskCommand.java:48) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.commands.StartTaskCommand.execute(StartTaskCommand.java:30) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.commands.CompositeCommand.execute(CompositeCommand.java:38) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.commands.TaskCommandExecutorImpl$SelfExecutionCommandService.execute(TaskCommandExecutorImpl.java:65) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.drools.core.command.impl.AbstractInterceptor.executeNext(AbstractInterceptor.java:41) [drools-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.persistence.TaskTransactionInterceptor.execute(TaskTransactionInterceptor.java:53) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.commands.TaskCommandExecutorImpl.execute(TaskCommandExecutorImpl.java:40) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.services.task.impl.command.CommandBasedTaskService.start(CommandBasedTaskService.java:245) [jbpm-human-task-core-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.console.ng.ht.backend.server.TaskServiceEntryPointImpl.startBatch(TaskServiceEntryPointImpl.java:306) [jbpm-console-ng-human-tasks-backend-6.0.1.Final.jar:6.0.1.Final]
at org.jbpm.console.ng.ht.backend.server.TaskServiceEntryPointImpl$Proxy$_$$_WeldClientProxy.startBatch(TaskServiceEntryPointImpl$Proxy$_$$_WeldClientProxy.java) [jbpm-console-ng-human-tasks-backend-6.0.1.Final.jar:6.0.1.Final]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_34]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_34]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_34]
at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_34]
at org.jboss.errai.bus.server.io.AbstractRPCMethodCallback.invokeMethodFromMessage(AbstractRPCMethodCallback.java:48) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.io.VoidRPCEndpointCallback.callback(VoidRPCEndpointCallback.java:20) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.io.RemoteServiceCallback.callback(RemoteServiceCallback.java:54) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.cdi.server.CDIExtensionPoints$3.callback(CDIExtensionPoints.java:499) [errai-weld-integration-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.DeliveryPlan.deliver(DeliveryPlan.java:47) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.ServerMessageBusImpl.sendGlobal(ServerMessageBusImpl.java:284) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.SimpleDispatcher.dispatchGlobal(SimpleDispatcher.java:46) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.service.ErraiServiceImpl.store(ErraiServiceImpl.java:92) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.service.ErraiServiceImpl.store(ErraiServiceImpl.java:109) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.errai.bus.server.servlet.DefaultBlockingServlet.doPost(DefaultBlockingServlet.java:140) [errai-bus-2.4.3.Final.jar:2.4.3.Final]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jboss.weld.servlet.ConversationPropagationFilter.doFilter(ConversationPropagationFilter.java:62) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jboss.solder.servlet.exception.CatchExceptionFilter.doFilter(CatchExceptionFilter.java:65) [solder-impl-3.2.1.Final.jar:3.2.1.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jboss.solder.servlet.event.ServletEventBridgeFilter.doFilter(ServletEventBridgeFilter.java:74) [solder-impl-3.2.1.Final.jar:3.2.1.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.jbpm.designer.web.filter.impl.PluggableFilter.doFilter(PluggableFilter.java:70) [jbpm-designer-backend-6.0.1.Final.jar:6.0.1.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.uberfire.security.server.UberFireSecurityFilter.doFilter(UberFireSecurityFilter.java:266) [uberfire-security-server-0.3.1.Final.jar:0.3.1.Final]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:]
at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:416) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_34]
-
3. Re: Processes getting stuck randomly for no apparent reason
swiderski.maciej Sep 3, 2014 1:59 AM (in response to zpirate)Zoltan,
both are thrown on start operation on the task and seems like both are caused by task not existing (first case) or not in the expected status (like it was already started). That might be a consequence of not properly refreshed task list in console after performing operation that resulted in having a task still present on the list with outdated operation options. That was found just before releasing 6.1 so there is high chance it was in 6.0.1 as well.
HTH
-
4. Re: Processes getting stuck randomly for no apparent reason
zpirate Sep 4, 2014 4:25 AM (in response to swiderski.maciej)Hi Maciej,
Does this mean that if we upgrade to 6.1, the problem should disappear? If it does happen again, is there a way to recover the processes that got stuck? Our processes are quite complex, they call many external services, so if they stop half way because of a lost internal event, they also leave external systems in an inconsistent state. It would be nice to have some trick that we could use to restore the process into a 'normal' state.
Thanks,
Zoltan
-
5. Re: Processes getting stuck randomly for no apparent reason
swiderski.maciej Sep 5, 2014 1:28 AM (in response to zpirate)I believe it's worth trying with 6.1 as indeed it might have been addressed there.
To recover from such situation you might want to use complete work item operation exposed by REST service that should move the process instance forward. Though it requires some manual operations to get the variables that were used when completing a task.
HTH