14 Replies Latest reply on Jun 15, 2012 5:28 AM by jena74

    Growing NioProcessor Thread Number in Human Tasks

    sandip.bhoi

      I am exploring the jBPM5.x and wanted to implement some work flows which mainly has Human iteractions and hence created Human Tasks.

      If I keep on executing perticular process definition, two NioProcessor Threads are getting created and remain in memory though I complete the Task and even complete entire process.

       

      e.g If my process definition has 5 Tasks and when I complete entire process, I can see 10 Threads got created and remains forever.

       

      Can any one tell why such thing is happening. I do not know why these threads are getting created and remain dangling.

        • 1. Re: Growing NioProcessor Thread Number in Human Tasks
          calca

          I don't know why it is happening, I think it could be a bug in human task mina connector. Could you make some thread dump to see these threads and what are they doing?.

          I recommend you using HornetQ connector, instead of mina one.

           

          Regards,

           

          Demian    

          1 of 1 people found this helpful
          • 2. Re: Growing NioProcessor Thread Number in Human Tasks
            sandip.bhoi

            Thanks Demian,

             

            Please find the Thread dump

             

            Thread Dump _____________57 - "http-bio-8080"-exec-9

            Thread Dump _____________63 - pool-22-thread-1

            Thread Dump _____________31 - H2 File Lock Watchdog C:\Project\POC\jbpm-installer\eclipse\JPADroolsFlow.lock.db

            Thread Dump _____________49 - "http-bio-8080"-exec-4

            Thread Dump _____________50 - "http-bio-8080"-exec-5

            Thread Dump _____________61 - pool-20-thread-1

            Thread Dump _____________20 - H2 File Lock Watchdog C:\Project\POC\jbpm-installer\eclipse\droolsflow.lock.db

            Thread Dump _____________13 - pool-1-thread-1

            Thread Dump _____________23 - Thread-11

            Thread Dump _____________3 - Finalizer

            Thread Dump _____________36 - NioProcessor-4

            Thread Dump _____________5 - Attach Listener

            Thread Dump _____________9 - GC Daemon

            Thread Dump _____________54 - NioProcessor-3

            Thread Dump _____________55 - pool-16-thread-1

            Thread Dump _____________39 - "http-bio-8080"-AsyncTimeout

            Thread Dump _____________40 - "http-bio-8080"-exec-1

            Thread Dump _____________52 - "http-bio-8080"-exec-7

            Thread Dump _____________48 - NioProcessor-7

            Thread Dump _____________1 - main

            Thread Dump _____________64 - NioProcessor-19

            Thread Dump _____________42 - "ajp-bio-8009"-AsyncTimeout

            Thread Dump _____________15 - pool-5-thread-1

            Thread Dump _____________18 - H2 TCP Server (tcp://10.136.4.198:9092)

            Thread Dump _____________21 - H2 Log Writer DROOLSFLOW

            Thread Dump _____________4 - Signal Dispatcher

            Thread Dump _____________32 - H2 Log Writer JPADROOLSFLOW

            Thread Dump _____________24 - bitronix-scheduler

            Thread Dump _____________56 - NioProcessor-10

            Thread Dump _____________14 - pool-3-thread-1

            Thread Dump _____________51 - "http-bio-8080"-exec-6

            Thread Dump _____________25 - NioSocketAcceptor-1

            Thread Dump _____________60 - "http-bio-8080"-exec-10

            Thread Dump _____________41 - "ajp-bio-8009"-Acceptor-0

            Thread Dump _____________37 - ContainerBackgroundProcessor[StandardEngine[Catalina]]

            Thread Dump _____________53 - "http-bio-8080"-exec-8

            Thread Dump _____________58 - pool-18-thread-1

            Thread Dump _____________45 - "http-bio-8080"-exec-3

            Thread Dump _____________46 - NioProcessor-2

            Thread Dump _____________59 - NioProcessor-13

            Thread Dump _____________10 - bitronix-disk-force-batcher

            Thread Dump _____________38 - "http-bio-8080"-Acceptor-0

            Thread Dump _____________62 - NioProcessor-16

            Thread Dump _____________2 - Reference Handler

            Thread Dump _____________44 - "http-bio-8080"-exec-2

            Thread Dump _____________30 - H2 TCP Server (tcp://10.136.4.198:9092) thread

             

            Printed by Code given below

             

            /**

                 *

                 */

                private static void printThreadDump() {

                    Map threadMap = Thread.getAllStackTraces();

                    for (Iterator threadMapIT = threadMap.keySet().iterator(); threadMapIT.hasNext();) {

                        Thread th = (Thread) threadMapIT.next();

                        //if (th.getName().contains("NioProcessor")) {

                            System.out.println("\nThread Dump _____________" + th.getId() + " - " + th.getName());

                        //}

                    }

                }

             

            Also I can see the logs in Tomcat

             

            SEVERE: The web application [/OrderFactoryWeb] appears to have started a thread named [NioProcessor-4] but has failed to stop it. This is very likely to create a memory leak.

            Nov 24, 2011 9:56:25 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

            SEVERE: The web application [/OrderFactoryWeb] appears to have started a thread named [NioProcessor-1] but has failed to stop it. This is very likely to create a memory leak.

            Nov 24, 2011 9:56:25 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

            SEVERE: The web application [/OrderFactoryWeb] appears to have started a thread named [H2 TCP Server (tcp://10.136.4.198:9092) thread] but has failed to stop it. This is very likely to create a memory leak.

            Nov 24, 2011 9:56:25 PM org.apache.coyote.AbstractProtocolHandler stop

            INFO: Stopping ProtocolHandler ["http-bio-8080"]

            Nov 24, 2011 9:56:25 PM org.apache.coyote.AbstractProtocolHandler stop

            INFO: Stopping ProtocolHandler ["ajp-bio-8009"]

            Nov 24, 2011 9:56:25 PM bitronix.tm.integration.tomcat55.BTMLifecycleListener lifecycleEvent

            INFO: Shutting down Bitronix Transaction Manager

            Nov 24, 2011 9:56:25 PM bitronix.tm.BitronixTransactionManager shutdown

            INFO: shutting down Bitronix Transaction Manager

            Nov 24, 2011 9:56:34 PM org.apache.mina.filter.logging.LoggingFilter log

            • 3. Re: Growing NioProcessor Thread Number in Human Tasks
              sandip.bhoi

              Also is this the normal behavior? My log is also keep growing at the interval if 10 sec even if every thing is idle.

               

              INFO: IDLE

              Nov 24, 2011 10:31:04 PM org.apache.mina.filter.logging.LoggingFilter log

              INFO: IDLE

              Nov 24, 2011 10:31:14 PM org.apache.mina.filter.logging.LoggingFilter log

              INFO: IDLE

              Nov 24, 2011 10:31:24 PM org.apache.mina.filter.logging.LoggingFilter log

              INFO: IDLE

              Nov 24, 2011 10:31:34 PM org.apache.mina.filter.logging.LoggingFilter log

              INFO: IDLE

              Nov 24, 2011 10:31:44 PM org.apache.mina.filter.logging.LoggingFilter log

              INFO: IDLE

              Nov 24, 2011 10:31:54 PM org.apache.mina.filter.logging.LoggingFilter log

              INFO: IDLE

              • 4. Re: Growing NioProcessor Thread Number in Human Tasks
                sandip.bhoi

                I have replaced MinaServer with HornetQ. But getting NPE as given below

                 

                SEVERE: Servlet /OrderFactoryWeb threw load() exception

                java.lang.NullPointerException

                    at org.jbpm.task.service.hornetq.HornetQTaskClientConnector.write(HornetQTaskClientConnector.java:165)

                    at org.jbpm.task.service.TaskClient.registerForEvent(TaskClient.java:565)

                    at org.jbpm.process.workitem.wsht.WSHumanTaskHandler.connect(WSHumanTaskHandler.java:69)

                    at com.sample.processserver.TaskProcessServlet.init(TaskProcessServlet.java:47)

                    at javax.servlet.GenericServlet.init(GenericServlet.java:160)

                    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1190)

                    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1103)

                    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1010)

                    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4957)

                    at org.apache.catalina.core.StandardContext$3.call(StandardContext.java:5284)

                    at org.apache.catalina.core.StandardContext$3.call(StandardContext.java:5279)

                    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

                    at java.util.concurrent.FutureTask.run(Unknown Source)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

                    at java.lang.Thread.run(Unknown Source)

                 

                Also if used CommandBasedWSHumanTaskHandler

                java.lang.NullPointerException

                    at org.jbpm.task.service.hornetq.HornetQTaskClientConnector.write(HornetQTaskClientConnector.java:165)

                    at org.jbpm.task.service.TaskClient.registerForEvent(TaskClient.java:565)

                    at org.jbpm.process.workitem.wsht.CommandBasedWSHumanTaskHandler.connect(CommandBasedWSHumanTaskHandler.java:93)

                    at com.sample.processserver.TaskProcessServlet.init(TaskProcessServlet.java:47)

                    at javax.servlet.GenericServlet.init(GenericServlet.java:160)

                    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1190)

                    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1103)

                    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1010)

                    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4957)

                    at org.apache.catalina.core.StandardContext$3.call(StandardContext.java:5284)

                    at org.apache.catalina.core.StandardContext$3.call(StandardContext.java:5279)

                    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

                    at java.util.concurrent.FutureTask.run(Unknown Source)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

                    at java.lang.Thread.run(Unknown Source)

                 

                What could be the wrong here?

                • 5. Re: Growing NioProcessor Thread Number in Human Tasks
                  calca

                  Hi,

                   

                  I did not have time to investigate Mina issue yet.

                   

                  About HornetQ issue,

                  it can be a problem when creating hornetq session.

                   

                  In this class:

                  https://github.com/droolsjbpm/jbpm/blob/5.1.x/jbpm-human-task/src/main/java/org/jbpm/task/service/hornetq/HornetQTaskClientConnector.java

                   

                  The NPE may indicate the session is null. Maybe the nor

                  But it should be created in connect method.

                   

                  -Could you check please if there is an error before? (line 122 of the class,

                  } catch (Exception e) {

                  logger.error(e.getMessage());

                  return false;

                  }

                   

                  )

                   

                  - If you don't see nothing wrong in logs, could you debug connect method to see if hornetq session is being created ok?

                   

                  Demian

                  • 6. Re: Growing NioProcessor Thread Number in Human Tasks
                    sandip.bhoi

                    Hi Demian,

                     

                    It is throwing at following highlighted line session = factory.createSession();

                    And Cause as NoSuchErrorMethod.

                     

                    Also I am using jbpm-human-task-5.1.0. version of jar.

                    CanNotCreateSession.JPG

                     

                    public boolean connect() {

                            if (session != null && !session.isClosed()) {

                                throw new IllegalStateException("Already connected. Disconnect first.");

                            }

                            try {

                                Map<String, Object> connectionParams = new HashMap<String, Object>();

                                if (address==null) {

                                    address = "127.0.0.1";

                                }

                                if (port==null) {

                                    port = 5445;

                                }

                                connectionParams.put(TransportConstants.PORT_PROP_NAME, port);

                                connectionParams.put(TransportConstants.HOST_PROP_NAME, address);

                     

                                TransportConfiguration transportConfiguration = new TransportConfiguration(NettyConnectorFactory.class.getCanonicalName(), connectionParams);

                                ClientSessionFactory factory = HornetQClient.createClientSessionFactory(transportConfiguration);

                                session = factory.createSession();

                                producer = session.createProducer(HornetQTaskServer.SERVER_TASK_COMMANDS_QUEUE);

                    • 7. Re: Growing NioProcessor Thread Number in Human Tasks
                      calca

                      Are you including these jars?

                       

                         <dependency>

                            <groupId>org.hornetq</groupId>

                            <artifactId>hornetq-core</artifactId>

                            <version>2.0.0.GA</version>

                          </dependency>

                          <dependency>

                            <groupId>org.hornetq</groupId>

                            <artifactId>hornetq-transports</artifactId>

                            <version>2.0.0.GA</version>

                          </dependency>

                          <dependency>

                            <groupId>javax.jms</groupId>

                            <artifactId>jms</artifactId>

                            <version>1.1</version>

                          </dependency>

                      <dependency>

                            <groupId>org.jboss.netty</groupId>

                            <artifactId>netty</artifactId>

                            <version>3.1.5.GA</version>

                          </dependency>

                       

                      Demian


                      1 of 1 people found this helpful
                      • 8. Re: Growing NioProcessor Thread Number in Human Tasks
                        sandip.bhoi

                        Thanks its passing further but stucked at taskSummaryHandler.getResults()

                         

                        // krisv is Product Planner

                            public static List<TaskSummary> getTaskDetails(String userName) {

                                List<TaskSummary> task1 = null;

                                try {

                                    SystemEventListenerFactory.setSystemEventListener(new SystemEventListener());

                                    TaskClient taskClient = createTaskClient();

                                    // "krisv" executes

                                    BlockingTaskSummaryResponseHandler taskSummaryHandler = new BlockingTaskSummaryResponseHandler();

                                    taskClient.getTasksAssignedAsPotentialOwner(userName, "en-UK", taskSummaryHandler);

                                    if (taskSummaryHandler != null && taskSummaryHandler.getResults() != null

                                            && taskSummaryHandler.getResults().size() > 0) {

                                        // task1 = taskSummaryHandler.getResults().get(0);

                                        task1 = taskSummaryHandler.getResults();

                                    }

                                    System.out.println("ProcessTest.getTaskDetails() task1:" + task1);

                         

                                    System.out.println("ProcessTest.getTaskDetails() ksession:" + ksession);

                         

                                } catch (Exception e) {

                                    System.out.println("ProcessTest.getTaskDetails()() Exception");

                                    e.printStackTrace();

                                }

                         

                                return task1;

                            }

                         

                        public List<TaskSummary> getResults() {

                                // note that this method doesn't need to be synced because if waitTillDone returns true,

                                // it means results is available

                                boolean done = waitTillDone(RESULTS_WAIT_TIME);

                         

                                if (!done) {

                                    throw new RuntimeException("Timeout : unable to retrieve results");

                                }

                         

                                return results;

                            }

                         

                         

                         

                         

                        Exception in thread "Thread-21" java.lang.RuntimeException: Client Exception with class class org.jbpm.task.service.hornetq.HornetQTaskClientConnector$1 using port 5445

                            at org.jbpm.task.service.hornetq.HornetQTaskClientConnector$1.run(HornetQTaskClientConnector.java:115)

                            at java.lang.Thread.run(Unknown Source)

                        Caused by: java.lang.ClassCastException: org.jbpm.process.workitem.wsht.CommandBasedWSHumanTaskHandler$TaskCompletedHandler cannot be cast to org.jbpm.task.service.TaskClientHandler$TaskSummaryResponseHandler

                            at org.jbpm.task.service.TaskClientHandler.messageReceived(TaskClientHandler.java:171)

                            at org.jbpm.task.service.hornetq.HornetQTaskClientHandler.messageReceived(HornetQTaskClientHandler.java:55)

                            at org.jbpm.task.service.hornetq.HornetQTaskClientConnector$1.run(HornetQTaskClientConnector.java:104)

                            ... 1 more

                        ProcessTest.getTaskDetails()() Exception

                        java.lang.RuntimeException: Timeout : unable to retrieve results

                            at org.jbpm.task.service.responsehandlers.BlockingTaskSummaryResponseHandler.getResults(BlockingTaskSummaryResponseHandler.java:41)

                            at com.sample.servlet.TaskWorkUtil.getTaskDetails(TaskWorkUtil.java:123)

                            at com.sample.servlet.OrderFactoryServlet.execute(OrderFactoryServlet.java:119)

                            at com.sample.servlet.OrderFactoryServlet.doGet(OrderFactoryServlet.java:41)

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)

                            at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

                            at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:297)

                            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

                            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

                            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)

                            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)

                            at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)

                            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)

                            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)

                            at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:563)

                            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

                            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:399)

                            at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:317)

                            at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:204)

                            at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:311)

                            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

                            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

                            at java.lang.Thread.run(Unknown Source)

                         

                         

                        ClassCastException.JPG

                         

                        Code files attached for reference

                        • 9. Re: Growing NioProcessor Thread Number in Human Tasks
                          calca

                          Hey,

                           

                          I am not sure what could be happenning, but, it could be caused because you are creating a new task client and hornetq connectors each time. So maybe the response is taken (in class HornetQTaskClientConnector) for some the last (or first?) connector (and thud task client handler) you created, and may lead to problems.

                           

                          Could you change your code to cache this task client in createHornetQTaskClient method for example and see what happens?

                           

                          Thanks

                           

                          Demian

                          • 10. Re: Growing NioProcessor Thread Number in Human Tasks
                            jena74

                            Hi,

                             

                            Did you succeed in solving this issue?

                            I am facing the same and I am blocked.

                            Thank you.

                             

                            Jena

                            • 11. Re: Growing NioProcessor Thread Number in Human Tasks
                              sandip.bhoi

                              No success...

                               

                              But eliminated the complete jBPM option... to fullfill my business requirement in the application implemented the raw code instead complex (may be due to learning curve) framework.

                              • 12. Re: Growing NioProcessor Thread Number in Human Tasks
                                jena74

                                Hello,

                                 

                                Thank you for the answer.

                                In fact, in the meantime I found the solution for me.

                                In my case, I was instanciating the HornetQTaskClientConnector with always the same name "client" (I mean the same value for the first parameter called name).

                                By luck, I saw an example of code somewhere using a different name for each HornetQTaskClientConnector by adding a randomly generated id to the "client" string.

                                Doing that the error disappeared and everything went well.

                                So for sure it is too late for you but I provide my answer in order maybe to help someone else.

                                 

                                On my side, even with this solved issue, I am still wondering if I am going to use jBPM or develop my own simple specific Java code because all these embedded frameworks are generating some strange errors really not easy to diagnose. I think nowadays there are too many frameworks embedded in frameworks or applications... and it is adding a real complexity to a lot of things. This complexity is usefull or not, this is the debate...

                                 

                                Jena

                                • 13. Re: Growing NioProcessor Thread Number in Human Tasks
                                  rajvij

                                  Hi Jena,

                                   

                                  Could you please provide the link of the example that you saw ?

                                  • 14. Re: Growing NioProcessor Thread Number in Human Tasks
                                    jena74

                                    Hi,

                                     

                                    In fact I am sorry but I lost the link.

                                    But, here is the Java code that I use now without error:

                                     

                                    String name = "client 1" + UUID.randomUUID();
                                    TaskClient client = new TaskClient(new HornetQTaskClientConnector(name, new HornetQTaskClientHandler(SystemEventListenerFactory.getSystemEventListener())));
                                    boolean isConnected = client.connect(HT_SERVICE_HOST, HT_SERVICE_PORT);
                                    System.out.println("isConnected = " + isConnected);

                                     

                                     

                                    It is working properly with jBPM 5.3.

                                     

                                    Jena