-
1. JMS client crash causes server deadlock?
clebert.suconic May 23, 2011 6:53 PM (in response to roarke100)Does it happen when you use the ConnectionFactory also, (it will only happen with the ThroughputConnectionFactory?)
-
2. Re: JMS client crash causes server deadlock?
roarke100 May 23, 2011 9:52 PM (in response to clebert.suconic)It happens with the ConnectionFactory as well. From what I have been able to dig into it seems to be a race condition during the connection closing process on the client. I haven't been able to pin-point where though. The server side is blocked trying to acquire credits (ClientProducerCreditsImpl.acquireCredits() line 67) but the client is long gone.
-
3. Re: JMS client crash causes server deadlock?
clebert.suconic May 23, 2011 9:57 PM (in response to roarke100)Can I ask you a favour before I jump into running your test?
Can you try either http://anonsvn.jboss.org/repos/hornetq/branches/Branch_2_2_EAP/ or http://anonsvn.jboss.org/repos/hornetq/trunk/
I remember some other issue I fixed around flow control.
-
4. Re: JMS client crash causes server deadlock?
roarke100 May 24, 2011 8:01 AM (in response to clebert.suconic)The issue still exists in the http://anonsvn.jboss.org/repos/hornetq/branches/Branch_2_2_EAP/ branch.
I also tested the trunk. I am still able to reproduce the problem there as well.
I will test more today, but it seems like there are two separate issues here I wasn't very clear in identifying before.
1) The client crashes at an inopportune time. I would expect at the very least at the time of the subscription timeout things should be cleared up properly and data should start flowing again to other subscriptions. This is what I initially found not to be the case. My test is setup to show this problem.
2) During the testing of item one, I found it was possible to get the server in the same state as mentioned at the top by just stepping through the subscription close logic in a debugger. This seems to imply to me there is a race condition in the close logic as well.
-
5. Re: JMS client crash causes server deadlock?
clebert.suconic May 24, 2011 5:29 PM (in response to roarke100)You are configuring your address to be Blocking.
You will have your producer blocked until you can start consuming again.
Notice: you configured it to block when your max-size-bytes achieved 10485760 bytes. Hence as soon as you stopped consuming, the server will block until you can resume it.
If you need paging: configure it to page.
Or even better: Restart your consumer after failing.
-
6. Re: JMS client crash causes server deadlock?
clebert.suconic May 24, 2011 5:30 PM (in response to clebert.suconic)Notice that Pre-ack will only ack the message when there's a consumer in place.
You could also set Time-to-live on the message. So the message leaves the memory (case you can lose the message in case of not being received)
-
7. Re: JMS client crash causes server deadlock?
clebert.suconic May 24, 2011 5:31 PM (in response to clebert.suconic)Just to finalize: I didn't find any issues: Please let me know if you I misunderstood anything from your config.
-
8. Re: JMS client crash causes server deadlock?
roarke100 May 24, 2011 5:45 PM (in response to clebert.suconic)How can I continue to consume the messages? These are unnamed non-durable subscriptions.
-
9. Re: JMS client crash causes server deadlock?
clebert.suconic May 24, 2011 6:56 PM (in response to roarke100)The credit should be returned once the connection goes beyond the ttl. The producer is not being unblocked after you see the message on connection dying? Are you seeing any message about the connection?
-
10. Re: JMS client crash causes server deadlock?
roarke100 May 24, 2011 9:39 PM (in response to clebert.suconic)That is correct. The connection goes beyond it's ttl and the messages remain blocked. I recieve the following message and the producer remains blocked:
2011-05-24 20:38:10,737 WARN [org.hornetq.core.server.impl.ServerSessionImpl] (
hornetq-failure-check-thread) Client connection failed, clearing up resources fo
r session 5d7ba212-866f-11e0-9fea-0026b97c87d9
2011-05-24 20:38:10,739 WARN [org.hornetq.core.server.impl.ServerSessionImpl] (
hornetq-failure-check-thread) Cleared up resources for session 5d7ba212-866f-11e
0-9fea-0026b97c87d9
It appears when the Queue is closed it does not free up the credits that are currently in the queue.
-
11. Re: JMS client crash causes server deadlock?
clebert.suconic May 24, 2011 11:53 PM (in response to roarke100)https://issues.jboss.org/browse/HORNETQ-700
I couldn't run your test yet.. I just looked at configs and found the blocking ... so I'm assuming this will be an issue.
Before I dig more into it (which will be some time this week), do you have your producer in one connection, and your consumer (with the temporary queue (i.e. the temporary subscription) in another connection?
-
12. Re: JMS client crash causes server deadlock?
roarke100 May 25, 2011 7:32 AM (in response to clebert.suconic)They are both use the same connection factory looked up via JNDI from two seperate JVMs.
-
13. Re: JMS client crash causes server deadlock?
clebert.suconic May 25, 2011 5:43 PM (in response to roarke100)I had fixed this on this morning.. I was just clearing it before I could commit it.
Thanks