clients stop receiving messages when client failure check period = 1 and one client does not exit cleanly

[hantunca]

In my testing of 2.1.0, I ran across a case where I have a server sending messages to multiple clients and one of the clients does not end cleanly, after a certain amount of time the clients that are still running stop receiving messages.  This started showing up when I set the client failure check period to -1.

I've attached a tar file that has the test case, source, etc.  To replicate the problem:

- start the server with server.sh.

- start a producer with producer.sh.

- start a client with client.sh.

- start a second client with client.sh - now with the client.sh, hit ctrl-c to kill the one client.

After a certain amount of time, the server will spit out the following message:

WARNING: Connection failure has been detected: Did not receive ping from /127.0.0.1:58343. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed. [code=3]

At this point, all clients stop receiving messages.  I would expect that the server would clean up the resources for the dead client and not stop messages from being sent to other clients.

This is not a big deal for me as I don't use the client failure check period, but happened to run across it during testing.

Han

[timfox]

What happens if you don't disable connection checking?

[hantunca]

If I don't disable it, clients continue to receive messages as expected.

by the way, noticed mad an error in the title of this thread - 'client failure check period = 1' should be 'client failure check period = -1'.

Han

[timfox]

I think this is to be expected.

By setting client-failure-check-period to -1 you're disabling the automatic cleanup of connections on client failure.

[hantunca]

Tim - I could understand not cleaning up resources at the server, i.e. causing memory leaks, resource usage, etc.  But should it cause messages to stop being sent to all other clients?  I have my server setup to send messages with no blocking, non-durable, pre-ack, so I'm expecting messages to continue to flow to the clients that are still connected.

thanks,

Han

[timfox]

What's happening here is because you are setting client-failure-check-period to -1, no pings ever get sent to the server, so the server will timeout both connections after connection ttl, irrespective of whether they've crashed or not.

If you set Configuration.clientttloverride to a large value, e.g. Long.MAX_VALUE / 2 you should prevent this. (Don't make it Long.MAX_VALUE or the calc will overflow)

However, it doesn't really make sense for you to disable client failure checking if your clients have a possibility of crashing, since you'll get resource leaks.