Lost messages with 150 concurrent requests for BPEL

[meghanai_99]

Hello,

Today I did some preliminary load testing on JBPM-BPEL on my dev machine. I used SoapUI to send 'n' number of concurrent requests. For requests upto 100, everything seemed ok. I have pointed my default DS to sql server database. As a result the internal messages from SoapHandler to StartListener go to my db where I can view them.

For small number after the load test completes, my queue is empty as expected. However when requests near 100 and above, there are some left over messages in the queue. First I got this error -

http://wiki.jboss.org/wiki/WhatDoesTheMessageNoManagedConnectionsAvailableMean

So I increased my connection pool size and also the waiting period for the threads. However even after doing this, I see left over messages in my queue. In my last run, I see 18 'connection timed out' errors from SoapUI out of 150 and 4 messages in the queue.

What do I do to improve this performance? Or I have to assume that this is a limitation? What happens to the messages left in the queue?

btw, after turning on trace statement in my ds file, I get warning from JBoss that it is closing result set that JBPM didn't close. I am running BPEL 1.1 GA.

Here is the stack trace -

a0001 WARN 2008-05-16 16:44:28,393 [WrappedConnection] -- [] Closing a result s
et you left open! Please close it yourself.
java.lang.Throwable: STACKTRACE
 at org.jboss.resource.adapter.jdbc.WrappedStatement.registerResultSet(Wr
appedStatement.java:617)
 at org.jboss.resource.adapter.jdbc.WrappedStatement.getGeneratedKeys(Wra
ppedStatement.java:533)
 at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.hibernate.util.GetGeneratedKeysHelper.getGeneratedKey(GetGenerate
dKeysHelper.java:69)
 at org.hibernate.id.IdentityGenerator$GetGeneratedKeysDelegate.executeAn
dExtract(IdentityGenerator.java:74)
 at org.hibernate.id.insert.AbstractReturningDelegate.performInsert(Abstr
actReturningDelegate.java:33)
 at org.hibernate.persister.entity.AbstractEntityPersister.insert(Abstrac
tEntityPersister.java:2093)
 at org.hibernate.persister.entity.AbstractEntityPersister.insert(Abstrac
tEntityPersister.java:2573)
 at org.hibernate.action.EntityIdentityInsertAction.execute(EntityIdentit
yInsertAction.java:47)
 at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:248)
 at org.hibernate.event.def.AbstractSaveEventListener.performSaveOrReplic
ate(AbstractSaveEventListener.java:290)
 at org.hibernate.event.def.AbstractSaveEventListener.performSave(Abstrac
tSaveEventListener.java:180)
 at org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId
(AbstractSaveEventListener.java:108)
 at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.saveWithGene
ratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:186)
 at org.hibernate.event.def.DefaultSaveEventListener.saveWithGeneratedOrR
equestedId(DefaultSaveEventListener.java:33)
 at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsTran
sient(DefaultSaveOrUpdateEventListener.java:175)
 at org.hibernate.event.def.DefaultSaveEventListener.performSaveOrUpdate(
DefaultSaveEventListener.java:27)
 at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpda
te(DefaultSaveOrUpdateEventListener.java:70)
 at org.hibernate.impl.SessionImpl.fireSave(SessionImpl.java:535)
 at org.hibernate.impl.SessionImpl.save(SessionImpl.java:523)
 at org.hibernate.impl.SessionImpl.save(SessionImpl.java:519)
 at org.jbpm.logging.db.DbLoggingService.log(DbLoggingService.java:50)
 at org.jbpm.svc.save.SaveLogsOperation.save(SaveLogsOperation.java:47)
 at org.jbpm.svc.Services.save(Services.java:164)
 at org.jbpm.JbpmContext.save(JbpmContext.java:427)
 at org.jbpm.bpel.integration.jms.StartListener.deliverRequest(StartListe
ner.java:229)
 at org.jbpm.bpel.integration.jms.StartListener.onMessage(StartListener.j
ava:168)
 at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java:696)
 at java.lang.Thread.run(Thread.java:595)

Thank you,
Meghana

[aguizar]

Certainly not a limitation, but an indication that you have to allocate more resources. JMS persistent messages, EJB timers and jBPM sessions all use database connections, possibly from different data sources. Does each data source have a pool large enough?

[meghanai_99]

Thanks for replying Alex.

I can allocate more resources but my main concern is why my messages are left in the queue. What happens to them after that? When a thread cannot get hold of any database connection, is that when the message is left in the queue? But I would imagine the StartListener receives the message first and then starts processing BPEL so message is already read. so I want to find out in which cases, the messages are just left in the queue.

btw, today I was trying to have more control over when listener picks up message so I tried JMS connection.stop(). According to API documentation it is supposed to stop the delivery of messages but that did not happen. I think this is not the right forum for that question but in case you know the answer, please let me know.

Thank you,
Meghana.

[aguizar]

The message is left in the queue when a non-recoverable exception occurs. Currently only StaleStateException and LockAcquisitionException are treated as recoverable. System exceptions, such a connection acquisition problem, need administrative action before proceeding, so they are not recovered automatically.

After you fix the problem you would recover the affected process manually. Neither the console nor a simple API call currently allow you to do so for individual process instances, but you can do it for all instances by redeploying the .ear.

Re: jmsConnection.stop(), there is a feature documented in the javadoc for that method which may explain the behavior:

If message listeners are running when stop is invoked, the stop call must wait until all of them have returned before it may return. While these message listeners are completing, they must have the full services of the connection available to them.

If no reception is in course when you invoke the method and still delivery does not stop, then there is an probably an anomaly in the implementation.