3 Replies Latest reply on May 23, 2008 1:11 PM by Alejandro Guizar

    Lost messages with 150 concurrent requests for BPEL

    Meghana Joglekar Novice

      Hello,

      Today I did some preliminary load testing on JBPM-BPEL on my dev machine. I used SoapUI to send 'n' number of concurrent requests. For requests upto 100, everything seemed ok. I have pointed my default DS to sql server database. As a result the internal messages from SoapHandler to StartListener go to my db where I can view them.

      For small number after the load test completes, my queue is empty as expected. However when requests near 100 and above, there are some left over messages in the queue. First I got this error -

      http://wiki.jboss.org/wiki/WhatDoesTheMessageNoManagedConnectionsAvailableMean

      So I increased my connection pool size and also the waiting period for the threads. However even after doing this, I see left over messages in my queue. In my last run, I see 18 'connection timed out' errors from SoapUI out of 150 and 4 messages in the queue.

      What do I do to improve this performance? Or I have to assume that this is a limitation? What happens to the messages left in the queue?

      btw, after turning on trace statement in my ds file, I get warning from JBoss that it is closing result set that JBPM didn't close. I am running BPEL 1.1 GA.

      Here is the stack trace -

      a0001 WARN 2008-05-16 16:44:28,393 [WrappedConnection] -- [] Closing a result s
      et you left open! Please close it yourself.
      java.lang.Throwable: STACKTRACE
       at org.jboss.resource.adapter.jdbc.WrappedStatement.registerResultSet(Wr
      appedStatement.java:617)
       at org.jboss.resource.adapter.jdbc.WrappedStatement.getGeneratedKeys(Wra
      ppedStatement.java:533)
       at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
      sorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.hibernate.util.GetGeneratedKeysHelper.getGeneratedKey(GetGenerate
      dKeysHelper.java:69)
       at org.hibernate.id.IdentityGenerator$GetGeneratedKeysDelegate.executeAn
      dExtract(IdentityGenerator.java:74)
       at org.hibernate.id.insert.AbstractReturningDelegate.performInsert(Abstr
      actReturningDelegate.java:33)
       at org.hibernate.persister.entity.AbstractEntityPersister.insert(Abstrac
      tEntityPersister.java:2093)
       at org.hibernate.persister.entity.AbstractEntityPersister.insert(Abstrac
      tEntityPersister.java:2573)
       at org.hibernate.action.EntityIdentityInsertAction.execute(EntityIdentit
      yInsertAction.java:47)
       at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:248)
       at org.hibernate.event.def.AbstractSaveEventListener.performSaveOrReplic
      ate(AbstractSaveEventListener.java:290)
       at org.hibernate.event.def.AbstractSaveEventListener.performSave(Abstrac
      tSaveEventListener.java:180)
       at org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId
      (AbstractSaveEventListener.java:108)
       at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.saveWithGene
      ratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:186)
       at org.hibernate.event.def.DefaultSaveEventListener.saveWithGeneratedOrR
      equestedId(DefaultSaveEventListener.java:33)
       at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsTran
      sient(DefaultSaveOrUpdateEventListener.java:175)
       at org.hibernate.event.def.DefaultSaveEventListener.performSaveOrUpdate(
      DefaultSaveEventListener.java:27)
       at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpda
      te(DefaultSaveOrUpdateEventListener.java:70)
       at org.hibernate.impl.SessionImpl.fireSave(SessionImpl.java:535)
       at org.hibernate.impl.SessionImpl.save(SessionImpl.java:523)
       at org.hibernate.impl.SessionImpl.save(SessionImpl.java:519)
       at org.jbpm.logging.db.DbLoggingService.log(DbLoggingService.java:50)
       at org.jbpm.svc.save.SaveLogsOperation.save(SaveLogsOperation.java:47)
       at org.jbpm.svc.Services.save(Services.java:164)
       at org.jbpm.JbpmContext.save(JbpmContext.java:427)
       at org.jbpm.bpel.integration.jms.StartListener.deliverRequest(StartListe
      ner.java:229)
       at org.jbpm.bpel.integration.jms.StartListener.onMessage(StartListener.j
      ava:168)
       at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java:696)
       at java.lang.Thread.run(Thread.java:595)
      


      Thank you,
      Meghana

        • 1. Re: Lost messages with 150 concurrent requests for BPEL
          Alejandro Guizar Master

          Certainly not a limitation, but an indication that you have to allocate more resources. JMS persistent messages, EJB timers and jBPM sessions all use database connections, possibly from different data sources. Does each data source have a pool large enough?

          • 2. Re: Lost messages with 150 concurrent requests for BPEL
            Meghana Joglekar Novice

            Thanks for replying Alex.

            I can allocate more resources but my main concern is why my messages are left in the queue. What happens to them after that? When a thread cannot get hold of any database connection, is that when the message is left in the queue? But I would imagine the StartListener receives the message first and then starts processing BPEL so message is already read. so I want to find out in which cases, the messages are just left in the queue.

            btw, today I was trying to have more control over when listener picks up message so I tried JMS connection.stop(). According to API documentation it is supposed to stop the delivery of messages but that did not happen. I think this is not the right forum for that question but in case you know the answer, please let me know.

            Thank you,
            Meghana.

            • 3. Re: Lost messages with 150 concurrent requests for BPEL
              Alejandro Guizar Master

              The message is left in the queue when a non-recoverable exception occurs. Currently only StaleStateException and LockAcquisitionException are treated as recoverable. System exceptions, such a connection acquisition problem, need administrative action before proceeding, so they are not recovered automatically.

              After you fix the problem you would recover the affected process manually. Neither the console nor a simple API call currently allow you to do so for individual process instances, but you can do it for all instances by redeploying the .ear.

              Re: jmsConnection.stop(), there is a feature documented in the javadoc for that method which may explain the behavior:

              If message listeners are running when stop is invoked, the stop call must wait until all of them have returned before it may return. While these message listeners are completing, they must have the full services of the connection available to them.

              If no reception is in course when you invoke the method and still delivery does not stop, then there is an probably an anomaly in the implementation.