5 Replies Latest reply on Aug 8, 2011 11:15 AM by tsurdilovic

    Guvnor stability

    tomarenz

      Hi, I'm trying to understand the source of an unstable behavior in Guvnor.

      Yesterday I was editing the (only one) process in the repository.Then I suspended my laptop.

      Today, after resuming,  I cannot proceed, since:

       

      - drools-guvnor, upon trying to list processes, says "Sorry, a technical error occurred. Please contact a system administrator". No process shown.

       

      - console, upon trying to list processes, waits for a while then it reports a request timeout. No process listed.

      Log is:

       

      2011-08-05 08:51:28,530 [DEBUG] New subscription: appContext.model.listener -> org.jboss.bpm.console.client.task.AssignedTasksView$7@11f
      2011-08-05 08:51:29,018 [DEBUG] GET: http://localhost:8080/gwt-console-server/rs/tasks/krisv
      2011-08-05 08:51:30,709 [DEBUG] parse {"tasks":[]}
      2011-08-05 08:51:30,716 [INFO ] Loaded 0 tasks
      2011-08-05 08:51:41,379 [DEBUG] ToolID: Process_Overview.4
      2011-08-05 08:51:41,492 [DEBUG] New subscription: appContext.model.listener > org.jboss.bpm.console.client.process.DefinitionListView$5@14b
      2011-08-05 08:51:42,029 [DEBUG] GET: http://localhost:8080/gwt-console-server/rs/process/definitions
      2011-08-05 08:52:02,035 [ERROR] <ul><li>URL: 'http://localhost:8080/gwt-console-server/rs/process/definitions'
      <li>Action: 'org.jboss.bpm.console.client.process.UpdateDefinitionsAction'
      <li>Exception: 'class java.io.IOException'</ul>

      Request timeout
      java.io.IOException:
      Request timeout
          at Unknown._Hb(Unknown source:0)
          at Unknown.xzc(Unknown source:0)
          at Unknown.cy(Unknown source:0)
          at Unknown.anonymous(Unknown source:0)
          at Unknown.HA(Unknown source:0)
          at Unknown.anonymous(Unknown source:0)
          at Unknown.anonymous(Unknown source:0)
      2011-08-05 08:52:21,498 [FATAL] Uncaught Exception:
      java.lang.IllegalArgumentException:
      must be positive

       

      Then I stopped/restarted jboss (ant stop.demo/ant start.demo). Same result afterwards.

      Is there any other log to look at ? Any other action I might take ?

      Thanks.

        • 1. Re: Guvnor stability
          tomarenz

          To provide further info, I looked at jboss log file server.log, after restarting it and running the console to get a process list.

          I see a couple of errors:

           

          2011-08-05 09:24:40,499 INFO  [STDOUT] (http-localhost%2F127.0.0.1-8080-5) ERROR 05-08 09:24:40,495 (RepositoryServlet.java:doAuthorizedAction:78)  
          java.lang.NullPointerException
          at java.io.OutputStream.write(OutputStream.java:58)
          ...

          2011-08-05 09:24:40,587 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/drools-guvnor].[PackageDeploymentServlet]]
            (http-localhost%2F127.0.0.1-8080-5) Servlet.service() for servlet PackageDeploymentServlet threw exception
          java.lang.NullPointerException
          at java.io.OutputStream.write(OutputStream.java:58)
          ...

          But the funny point is that now, after an hour of attempts, a process appears both in drools-guvnor and the console !

          But in drools-guvnor is my new Test process, while in console is Evaluation.

          • 2. Re: Guvnor stability
            tsurdilovic

            jBPM-console currently looks for assets in the pkg file of the Guvnor defaultPackage only (we are working currently on expading the lookup to all available packages in Guvnor). Unless you build your package in Guvnor after making changes, console will not be able to see your changes. It could be that you have not built your package and the no-longer existing Evaluation process is still present in the pkg?

             

            Both Guvnor and jBPM-console are webapps and have a session timeout. After a long absence, you would need to re-log back into guvnor (I think the error "Sorry, a technical error occurred. Please contact a system administrator" is indicating this, you would more accurate error in the server log). This is probably also the reason for the other errors but I am not sure without seeing the entire stack of the logs.

             

            Hope this helps.

            • 3. Re: Guvnor stability
              tomarenz

              I'm afraid it isn't so easy.

              After the weekend (and yes, logging in again) I cannot reopen my test process. I got an out-of-memory server error, and the jboss server.log reported below (I apologize for the long report).

              Finally I stopped/restarted jboss then guvnor, my test project is there but totally empty.

               

              java.lang.reflect.InvocationTargetException

                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                  at java.lang.reflect.Method.invoke(Method.java:597)

                  at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)

                  at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:31)

                  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56)

                  at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28)

                  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)

                  at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:77)

                  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)

                  at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44)

                  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)

                  at org.jboss.seam.security.SecurityInterceptor.aroundInvoke(SecurityInterceptor.java:157)

                  at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)

                  at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)

                  at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:166)

                  at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:102)

                  at org.drools.guvnor.server.RepositoryPackageService_$$_javassist_8.buildPackage(RepositoryPackageService_$$_javassist_8.java)

                  at org.drools.guvnor.server.RepositoryServiceServlet.buildPackage(RepositoryServiceServlet.java:282)

                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                  at java.lang.reflect.Method.invoke(Method.java:597)

                  at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:569)

                  at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)

                  at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)

                  at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)

                  at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

                  at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

                  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

                  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

                  at org.jboss.seam.web.ContextFilter$1.process(ContextFilter.java:42)

                  at org.jboss.seam.servlet.ContextualHttpServletRequest.run(ContextualHttpServletRequest.java:53)

                  at org.jboss.seam.web.ContextFilter.doFilter(ContextFilter.java:37)

                  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

                  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

                  at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

                  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

                  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

                  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)

                  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

                  at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)

                  at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)

                  at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)

                  at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)

                  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

                  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

                  at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)

                  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

                  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)

                  at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)

                  at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)

                  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)

                  at java.lang.Thread.run(Thread.java:619)

              Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

              2011-08-08 09:59:29,321 WARN  [org.jboss.system.server.profileservice.hotdeploy.HDScanner] (HDScanner) Scan failed

              java.lang.OutOfMemoryError: GC overhead limit exceeded

              2011-08-08 09:59:47,131 INFO  [STDOUT] (http-localhost%2F127.0.0.1-8080-6) (null: -1, -1): Premature end of file.

              2011-08-08 09:59:56,618 INFO  [org.oryxeditor.server.EditorHandler] (http-localhost%2F127.0.0.1-8080-6) Performing diagram information pre-processing steps.

              2011-08-08 10:00:03,875 ERROR [com.intalio.web.preprocessing.impl.JbpmPreprocessingUnit] (http-localhost%2F127.0.0.1-8080-6) Read timed out

              2011-08-08 10:00:05,046 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/designer].[EditorHandler]] (http-localhost%2F127.0.0.1-8080-6) Servlet.service() for servlet EditorHandler threw exception

              java.lang.OutOfMemoryError: GC overhead limit exceeded

              2011-08-08 10:00:06,236 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/drools-guvnor]] (http-localhost%2F127.0.0.1-8080-8) StandardWrapper.Throwable

              java.lang.NoClassDefFoundError: Could not initialize class org.apache.cxf.jaxrs.utils.JAXRSUtils

                  at org.apache.cxf.jaxrs.model.ClassResourceInfo.getConsumeMime(ClassResourceInfo.java:239)

                  at org.apache.cxf.jaxrs.model.OperationResourceInfo.checkMediaTypes(OperationResourceInfo.java:167)

                  at org.apache.cxf.jaxrs.model.OperationResourceInfo.<init>(OperationResourceInfo.java:74)

                  at org.apache.cxf.jaxrs.utils.ResourceUtils.createOperationInfo(ResourceUtils.java:337)

                  at org.apache.cxf.jaxrs.utils.ResourceUtils.evaluateResourceClass(ResourceUtils.java:214)

                  at org.apache.cxf.jaxrs.utils.ResourceUtils.createClassResourceInfo(ResourceUtils.java:200)

                  at org.apache.cxf.jaxrs.JAXRSServiceFactoryBean.createResourceInfo(JAXRSServiceFactoryBean.java:186)

                  at org.apache.cxf.jaxrs.JAXRSServiceFactoryBean.setResourceClasses(JAXRSServiceFactoryBean.java:150)

                  at org.apache.cxf.jaxrs.JAXRSServerFactoryBean.setResourceClasses(JAXRSServerFactoryBean.java:167)

                  at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.loadBus(CXFNonSpringJaxrsServlet.java:97)

                  at org.apache.cxf.transport.servlet.AbstractCXFServlet.init(AbstractCXFServlet.java:84)

                  at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1048)

                  at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:777)

                  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:129)

                  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

                  at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)

                  at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)

                  at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)

                  at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)

                  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

                  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

                  at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)

                  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

                  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)

                  at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)

                  at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)

                  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)

                  at java.lang.Thread.run(Thread.java:619)

              2011-08-08 10:00:07,430 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/drools-guvnor].[CXFServlet]] (http-localhost%2F127.0.0.1-8080-8) Allocate exception for servlet CXFServlet

              java.lang.NoClassDefFoundError: Could not initialize class org.apache.cxf.jaxrs.utils.JAXRSUtils

                  at org.apache.cxf.jaxrs.model.ClassResourceInfo.getConsumeMime(ClassResourceInfo.java:239)

                  at org.apache.cxf.jaxrs.model.OperationResourceInfo.checkMediaTypes(OperationResourceInfo.java:167)

                  at org.apache.cxf.jaxrs.model.OperationResourceInfo.<init>(OperationResourceInfo.java:74)

                  at org.apache.cxf.jaxrs.utils.ResourceUtils.createOperationInfo(ResourceUtils.java:337)

                  at org.apache.cxf.jaxrs.utils.ResourceUtils.evaluateResourceClass(ResourceUtils.java:214)

                  at org.apache.cxf.jaxrs.utils.ResourceUtils.createClassResourceInfo(ResourceUtils.java:200)

                  at org.apache.cxf.jaxrs.JAXRSServiceFactoryBean.createResourceInfo(JAXRSServiceFactoryBean.java:186)

                  at org.apache.cxf.jaxrs.JAXRSServiceFactoryBean.setResourceClasses(JAXRSServiceFactoryBean.java:150)

                  at org.apache.cxf.jaxrs.JAXRSServerFactoryBean.setResourceClasses(JAXRSServerFactoryBean.java:167)

                  at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.loadBus(CXFNonSpringJaxrsServlet.java:97)

                  at org.apache.cxf.transport.servlet.AbstractCXFServlet.init(AbstractCXFServlet.java:84)

                  at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1048)

                  at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:777)

                  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:129)

                  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

                  at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)

                  at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)

                  at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)

                  at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)

                  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

                  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

                  at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)

                  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

                  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)

                  at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)

                  at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)

                  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)

                  at java.lang.Thread.run(Thread.java:619)

              • 4. Re: Guvnor stability
                tomarenz

                The overall sequence can be reproduced, by creating a process such as:

                 

                start/initial task/split xor gateway/two branches, one task each/join gateway/final task/end.

                 

                No properties or names. Save and close. Open again, then the jboss outOfMemory error appears.

                • 5. Re: Guvnor stability
                  tsurdilovic

                  I'm not able to reproduce your

                  Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

                  error. IIRC in these cases you should make sure your heap is big enough and possible also try to turn off the overhead feature with

                  -XX:-UseGCOverheadLimit