10 Replies Latest reply on Oct 11, 2009 6:42 AM by timfox

How fast are stale connections closed?

mr_dronski Sep 25, 2009 12:13 PM

I'm observing a problem with a remote hornetq broker (linux + libaio, but journal change didn't affect anything).

Basically, a single consumer is using the queue, but if this consumer (it's a message listener, not poller) is killed, then Server->JMS->listRemoteAdresses() lists 16 connections (I ensured it was 1 connection from my app only).

Next, the problem is consumer won't be able to get any new messages until after these stale connection refs are removed (I guess that's a connection limit on the connection factory, but it is set to 8 by defaul>t).

The consumer is non-TX, non-XA.

Any hints on how to diagnose this leak?

1. Re: How fast are stale connections closed?

timfox Sep 25, 2009 12:14 PM (in response to mr_dronski)

See http://hornetq.sourceforge.net/docs/hornetq-2.0.0.BETA5/user-manual/en/html/connection-ttl.html
Actions
2. Re: How fast are stale connections closed?

mr_dronski Sep 25, 2009 12:14 PM (in response to mr_dronski)

Another data point - this setup always worked fine on a win xp machine locally, things manifested themselves after having been moved to a remote linux box (consumer is on the win box).
Actions
3. Re: How fast are stale connections closed?

mr_dronski Sep 25, 2009 12:17 PM (in response to mr_dronski)

If I just connect the consumer, the jmx console lists 2 jms connections, albeit it's only 1 from my app. Every time I kill the consumer (deliberately, need to test reliability), the connections are left open, and next startup add another 2. This is without any messages on the queue, it's empty.

When queue is loaded, the connection count jumps to 16.
Actions
4. Re: How fast are stale connections closed?

timfox Sep 25, 2009 12:18 PM (in response to mr_dronski)

if the server shows 16 connections, then *somehow* your client has created 16 connections.

Perhaps the client is creating a connection then exiting without closing it cleanly.

In this case the connection will still alive until connection ttl (see link I posted in the previous reply) when it will be cleaned up.
Actions
5. Re: How fast are stale connections closed?

mr_dronski Sep 25, 2009 12:34 PM (in response to mr_dronski)

Thanks for the link, Tim. Let's focus on connection cleanup for now and ignore the rest. My point is the server must regularly do connection swipes and cleanup dead ones, 5 minutes could be too long a wait maybe? I'm specifically pulling the network plug (yeah, I'm bad :) to see if and how fast I can break the server). With a 5 minute default this makes the DoS attack so much easier.
Actions
6. Re: How fast are stale connections closed?

timfox Sep 25, 2009 12:38 PM (in response to mr_dronski)

Sure, it's completely configurable.

If 5 mins is too long for you, you can make it shorter.
Actions
7. Re: How fast are stale connections closed?

timfox Sep 25, 2009 12:38 PM (in response to mr_dronski)

Actually, now you mention it.

I'm going make the default shorter anyway in the next release.
Actions
8. Re: How fast are stale connections closed?

mr_dronski Sep 25, 2009 12:42 PM (in response to mr_dronski)

Thanks.
Actions
9. Re: How fast are stale connections closed?

dtauzell Oct 9, 2009 3:45 PM (in response to mr_dronski)

What concerns me is that if I:

1. Put messages onto the queue
2. Application crashes before closing connections

now, any new consumers cannot receive the messages until the 5 minute limit is up. Why is that?

-Dave
Actions
10. Re: How fast are stale connections closed?

timfox Oct 11, 2009 6:42 AM (in response to mr_dronski)

"dtauzell" wrote:
What concerns me is that if I:

1. Put messages onto the queue
2. Application crashes before closing connections

now, any new consumers cannot receive the messages until the 5 minute limit is up. Why is that?

-Dave

See chapter 17 http://hornetq.sourceforge.net/docs/hornetq-2.0.0.BETA5/user-manual/en/html/connection-ttl.htm

You can always set connection ttl to a lower value if you like. (Actually it's already lower in TRUNK)
Actions

Go to original post