MDB Fails to retrieve messages from Topic after bombardment

[pallidan]

First off, I am very new to jboss, and the system was setup when I got here, so I'll do my best to give more information, but directions on where to look would be much appreciated.

Basically, we have a system that uses JBossMQ and Topics. Messages are given to a topic and a couple of MDBs process the messages. What I have found is that if I send many messages at once to the topic, some MDBs will choke. Basically they'll start processing messages but then they'll just stop and not retrieve any more unless I restart the bean. If an MDB has to do a lot of processing, the number of messages it can handle before this happens decreases dramatically.

We had it set up so MaximumSize=1 and MaxMessages=1. We found that by increasing MaximumSize, it made the system more stable in that it would last longer before choking, however we would lose ordering, which is desired.

I'm not sure where else to start reading and looking. I have found postings on java's help pages describing the same problem and the only help posted was "this sounds like a jboss bug" and "try switching to ActiveMQ and see if that helps", but my investigation leads me to believe that switching to ActiveMQ would be non-trivial.

We're using jboss 3.2.1, Windows 2000, and I don't expect it to be upgrade any time in the immediate future. I tried adding some tracing on the MDB and found that after the last message was handled before the bean stopped reading messages was this:

12:06:54,875 TRACE [StdServerSession] onMessage done
12:06:54,875 TRACE [StdServerSessionPool] using server session: org.jboss.jms.as
f.StdServerSession@3aec42
12:06:54,890 TRACE [StdServerSession] starting invokes on server session
12:06:54,890 TRACE [StdServerSessionPool] getting a server session
12:06:54,890 TRACE [StdServerSession] running...

So it's starting, but it just doesn't get around to actually reading.

Any help or places for more information would be appreciated. I have read through the forums to no avail, the faqs that I've found and the manual that I have. But, I may have overlooked things because of my newbieness.

Thanks a lot.

[pallidan]

I should mention that if this is a known bug that has been fixed, I may be able to make a case to upgrade to a newer version of jboss.

Thanks.

[genman]

Hmm, I would at least try to switch to 3.2.7. There's been many fixed bugs.

[adrian.brock]

And this is a bug in oswego-concurrent anyway :-)

[thoennes]

In this JBoss version, oswego-concurrent thread pools are used to process the messages. Every message picked up is handed over to the thread pool for processing.

If the thread pool is set to 1 (using maximumSize), the MDB acts as a singleton.

If some reasons a thread from the thread pools exits (due to an exception etc), in some cases no thread is re-created by the thread pool. So you have some sort of thread leakage. Restarting the MDB probably re-creates the thread pool completely.

We also use singleton MDBs and had a similiar looking case a while ago. From the java thread dump we could see that the blocking MDB was waiting on its thread pool for a thread to process the message. But the pool was empty, so the MDB waited forever.

Dough Lea, the creator of the oswego-concurrent package, said the reason was an unsupported way how JBoss used the thread pool settings, but Adrian regarded it as a bug in the lib.

Anyway, Dough added a change to support the JBoss usage and now it works.

Just have a look at the thread dump and upgrade the version of the concurrent library you are using. The latest version you can find here:

http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/intro.html

Replace the version used by JBoss and try again.

Upgrading to the newest JBoss in the 3.2 branch is a good idea anyway, besides of pulling in the newest concurrent lib it would also fix a couple of other issues with JMS.

HTH. Jörg

[pallidan]

Thanks for your help. I tried the new concurrent.jar that came with jboss 3.2.8, but it didn't fix the problem. In fact, whiel the problem still occured, the event handling seemed to be out of order.

I'll have to try switching completely to 3.2.8 when I have the chance. When I get around to it, I'll post my results!

Thanks a lot!

[adrian.brock]

"Thoennes" wrote:

Dough Lea, the creator of the oswego-concurrent package, said the reason was an unsupported way how JBoss used the thread pool settings, but Adrian regarded it as a bug in the lib.

It is a bug. Just because Doug no longer supports it, doesn't stop it being a bug. ;-)

It is still a bug as well in the latest oswego concurrent, we just have a workaround
that avoids the problem (we don't let the threadpool go empty through idle removal).

The problem comes because you have something like:

Scheduling thread: Are there any unused threads? Yes
Idle Remover: Remove all the unused threads (they just timed out)
Scheduling thread: Add the work for the now non-existant threads to pickup
Result: Work is never performed

It is a simple race condition. The check for unused threads and the addition of work
are not synchronized (atomic).
It is also NOT easy to fix because of the design of the code. Which is why Doug
does not support it.

The handling of idle threads in general is pretty poor in concurrent's executor. It is very
difficult to achieve a controlled shutdown that doesn't leaving hanging/orphaned
threads lying around. Again NOT easy to fix.

[adrian.brock]

It is very
difficult to achieve a controlled shutdown that doesn't leaving hanging/orphaned
threads lying around.

To be later removed when they pass the idle timeout ("long after" the thread pool is dead).