-
1. Re: 3.2.2RC3, Fiorano, MDB, concurrency
adrian.brock Sep 1, 2003 11:22 AM (in response to gavinandrews)Start with looking for errors in server.log
Enabling TRACE debugging for org.jboss.ejb and org.jboss.jms
might show more information?
Also check for a deadlock by dumping the threads.
Regards,
Adrian -
2. Re: 3.2.2RC3, Fiorano, MDB, concurrency
gavinandrews Sep 2, 2003 8:06 AM (in response to gavinandrews)I turned debugging on; It just seems that after a bit of concurrency the onMessage of my MDBs are not being invoked.
The problem is easily reproduceable; just take a trivial MDB and put some debugging and a Thread.Sleep in the onMessage. Then hammer the server with lots of messages; after a while it gets stuck and despite the JMS queue having work; JBoss no longer takes the messages off the queue.
I have tried all combinations of Bean/Container and Required/NotSupported but get the same behaviour.
processing can be resumed by stopping and restarting the JMSProvider..
As I say I'm using FioranoMQ 5.21 but I can't see why this behaviour would necessarily be Provider specific?
Anything else I can try?
I looked in JMSContainerInvoker and MessageDrivenMetaData but I can't see anything obvious... the comment about ClientAcknowledge seems a odd but I don't think that's my problem.
Regards,
Gavin -
3. Re: 3.2.2RC3, Fiorano, MDB, concurrency
adrian.brock Sep 2, 2003 8:17 AM (in response to gavinandrews)The delivery goes something like:
MDB Container -> Fiorano ConnectionConsumer to get message
MDB Conatiner -> ServerSessionPool -> FioranoSession
FioranoSession -> JBoss MDB invoker -> your MDB
Now the question is where is it getting stuck?
Can you post a threaddump and the log?
Regards,
Adrian -
4. Re: 3.2.2RC3, Fiorano, MDB, concurrency
gavinandrews Sep 2, 2003 9:07 AM (in response to gavinandrews)I don't have a thread dump or log...
I found a potential difference in how I did things before using JBOSS MDB... previously I have created non-transacted QueueConnections.
I see from JMSContainerInvoker that the pool is always created as transacted (there is 'true' hardcoded)...
Remember this is FioranoMQ 5.21 so all my sessions are non-XA.
I wonder if this has something to do with it? Incidentally how do you turn trace on to see when the sessions are pullled from the pool?
Thanks,
Gavin -
5. Re: 3.2.2RC3, Fiorano, MDB, concurrency
adrian.brock Sep 2, 2003 9:18 AM (in response to gavinandrews)It should detect automatically that it is not XA
You should have the following warning in your log:
log.warn("Using a non-XA TopicConnection. " +
"It will not be able to participate in a Global UOW");
There is an example for TRACE in conf/log4j.xml
Regards,
Adrian -
6. Re: 3.2.2RC3, Fiorano, MDB, concurrency
gavinandrews Sep 2, 2003 9:26 AM (in response to gavinandrews)I do get the...
"Using a non-XA Queue Connection. It will not..."
messages; and the sessions are being created in the pool...
"added server session to the pool"
What I am suspicious about is the .createQueueSession(transacted, ack) since transacted is set true from JMSContainerInvoker and in this case I don't have a transaction.
Regards,
Gavin -
7. Re: 3.2.2RC3, Fiorano, MDB, concurrency
adrian.brock Sep 2, 2003 9:32 AM (in response to gavinandrews)It is a JMS client transaction, nothing to do with JTA.
Regards,
Adrian -
8. Re: 3.2.2RC3, Fiorano, MDB, concurrency
gavinandrews Sep 2, 2003 9:39 AM (in response to gavinandrews)>>It is a JMS client transaction, nothing to do with JTA.
I can see that.
I'll turn on tracing and try it again...
Incidentally... stopDelivery, then start and then startDelivery on the bean gets things going again... for a while...
Regards,
Gavin -
9. Re: 3.2.2RC3, Fiorano, MDB, concurrency
gavinandrews Sep 3, 2003 2:57 AM (in response to gavinandrews)I had a brainwave...
Instead of being related to concurrency this issue is simply competition for the StdJMSPool. I then reduced the StdJMSPool MaxSize to 2 and started JBOSS.
The first message is consumed correctly...
The second message is consumed correctly...
and the 3rd hangs the server.
Looks like sessions are not being returned to the pool or aren't re-useable or something like that...?
Regards,
Gavin -
10. Re: 3.2.2RC3, Fiorano, MDB, concurrency
gavinandrews Sep 3, 2003 3:04 AM (in response to gavinandrews)Yeap; that's it...
The first message works ok...
The second message works ok...
Then when the third message arrives I get
2003-09-03 09:52:00,599 TRACE [org.jboss.jms.asf.StdServerSessionPool] getting a server session
and then it stops... no more messages... no nothing.
Regards,
Gavin -
11. Re: 3.2.2RC3, Fiorano, MDB, concurrency
gavinandrews Sep 3, 2003 3:16 AM (in response to gavinandrews)Here is a slightly more complete trace... i.e. it shows the 2nd message arriving and being serviced...
2003-09-03 09:51:20,146 TRACE [org.jboss.jms.asf.StdServerSessionPool] getting a server session
2003-09-03 09:51:20,146 TRACE [org.jboss.jms.asf.StdServerSessionPool] using server session: org.jboss.jms.asf.StdServerSession@1d652
2003-09-03 09:51:20,146 TRACE [org.jboss.jms.asf.StdServerSession] starting invokes on server session
2003-09-03 09:51:20,146 TRACE [org.jboss.jms.asf.StdServerSession] onMessage running (pool, session, xaSession, useLocalTX): , fiorano.jms.runtime.ptp.FioranoQueueSession@19e3e24, null, false
2003-09-03 09:51:20,146 TRACE [org.jboss.ejb.plugins.jms.JMSContainerInvoker] processing message: fiorano.jms.services.msg.def.FioranoTextMessage@4d6b1a
2003-09-03 09:51:20,146 TRACE [org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor] invokerBInding is null in ProxyFactoryFinder
2003-09-03 09:51:20,146 TRACE [org.jboss.ejb.plugins.LogInterceptor] Start method=onMessage
2003-09-03 09:51:20,146 TRACE [org.jboss.ejb.plugins.MessageDrivenInstancePool] Get instance org.jboss.ejb.plugins.MessageDrivenInstancePool@11c135c#1#class MDB
2003-09-03 09:51:20,146 INFO [STDOUT] >onMessage
2003-09-03 09:51:20,146 INFO [STDOUT] Bean got message from onMessage():09:51:20 Test
2003-09-03 09:51:20,146 INFO [STDOUT] Starting Wait...
2003-09-03 09:51:20,161 TRACE [org.jboss.jms.asf.StdServerSession] running...
2003-09-03 09:51:22,146 INFO [STDOUT] Finished Waiting
2003-09-03 09:51:22,146 INFO [STDOUT] <onMessage
2003-09-03 09:51:22,146 TRACE [org.jboss.ejb.plugins.MessageDrivenInstancePool] 0/100 Free instance:org.jboss.ejb.plugins.MessageDrivenInstancePool@11c135c#null#null#true#class MDB
2003-09-03 09:51:22,146 TRACE [org.jboss.ejb.plugins.LogInterceptor] End method=onMessage
2003-09-03 09:51:22,146 TRACE [org.jboss.jms.asf.StdServerSession] Commiting the JMS transaction
2003-09-03 09:51:22,146 TRACE [org.jboss.jms.asf.StdServerSession] onMessage done
2003-09-03 09:51:40,489 TRACE [org.jboss.ejb.plugins.LRUEnterpriseContextCachePolicy] Resized cache for bean InstrumentPrice: old capacity = 1000000, new capacity = 50
<<<<<<<<< 3rd message arrives
2003-09-03 09:52:00,599 TRACE [org.jboss.jms.asf.StdServerSessionPool] getting a server session
2003-09-03 09:52:10,411 TRACE [org.jboss.ejb.plugins.LRUEnterpriseContextCachePolicy] Resized cache for bean BorisOrderProperty: old capacity = 1000000, new capacity = 50
I can't see the sessions being recycled into StdJMSPool; which I guess explains why they get used up and then it it gets blocked waiting on the SessionPool.
Regards,
Gavin -
12. Re: 3.2.2RC3, Fiorano, MDB, concurrency
adrian.brock Sep 3, 2003 4:20 AM (in response to gavinandrews)So the Fiorano session is not returning from its run() operation?
Regards,
Adrian -
13. Re: 3.2.2RC3, Fiorano, MDB, concurrency
gavinandrews Sep 3, 2003 5:22 AM (in response to gavinandrews)That appears to be it.
Fiorano's 5.21 implementation of..
javax.jms.Session.run for the created QueueSession does not apper to return once all outstanding messages have been sent to the session and hence
the StdServerSession.run doesn't move on and hence the session is not recycled.
Not sure what to do now... might just go and bang my head against a wall.
Regards,
Gavin -
14. Re: 3.2.2RC3, Fiorano, MDB, concurrency
adrian.brock Sep 3, 2003 5:44 AM (in response to gavinandrews)Take a threaddump (ctrl-break windows, ctrl-\ on unix)
Look for deadlocks.
Regards,
Adrian