clientLeasePeriod for JMS clients (bisocket)

[gsanderson]

We recently upgraded from JBoss 4.05 to 4.3, and moved to the new JBoss messaging stack.

In the simplest case, we have a bunch of clients subscribed to a JMS topic listening for messages from an un-clustered server.

We noticed while debugging the client that it would suddenly cease to receive JMS events from the server, and we have also seen this problem with VMs and OS X clients.

It seems that the cause of at least the debugging problem (and possibly since the symptoms match the others - though it is unclear thus far what would be delaying the pings significantly) that the default 10s lease period is the problem.

My question therefore when deciding how much to up this by, is how critical is the protection provided by this short lease time.

1) Obviously a dead client is wasting some server side book-keeping, however it would seem that a subsequent attempt to write a message to the topic, gets a socket write error if the client is dead, dumps a stack trace on the server console, and that particular subscriber is not hit again
2) I have seen other threads which suggest that the publisher can get blocked when writing to some more insidiously dead clients, and perhaps the lease is a protection against that.

Note, w.r.t. 1) after the subscriber is removed, I still see a later WARN message about the lease timing out, so it seems like the two forms of clean up are different.

Anyway, again, the first question is "is upping the lease to a few minutes a reasonable thing to do?"

Secondly, we wouldn't really care so much if we lost the lease if there was a clean way to detect it from the client and re-subscribe - is there any form of callback we can attach to when the client notices that its connection has died?

Thirdly, if there isn't such a way, we also have ways to detect this, but would be curious to know whether we're going to end up leaking stuff on the client because the existing subscriber mechanism on the client is just stuck there.

[gsanderson]

Actually, while this is likely my problem for the debugging case, I believe that it is much more likely that in the other cases, the client closes the connection.

This happens because there is a ConnectionValidator associated with the JMS connection (not sure the exact details yet), and this is pinging every 2ms.

I don't think the ping frequency is even the real problem (though it seems alarmingly high) I think the problem is that the SO_TIMEOUT used when checking the connection is only 1 second !!!! (note this is in the jboss remoting code)

Is this really the expected behavior? Your JMS connection gets closed if pinging the server takes more than a second?

[gsanderson]

In case anyone else has this problem, i did some more digging and it seems that the latter problem (the unchangeable 1s timeout in jboss remoting) should be fixed by the following

https://jira.jboss.org/jira/browse/JBREM-755

https://jira.jboss.org/jira/browse/JBREM-1069

(the latter of which is exceedingly recent and not yet released).

I'll post the results when I get a chance to play with that build.

[timfox]

You don't mention what version of JBM you are using. But in the latest version, the default ping is already changed to a more sensible value.

[gsanderson]

Sorry, we're using whatever comes with

JBoss 4.3.0EAP CP 03

I'm guessing that the ping requires a Java level response, so we should consider (potentially badly configured) GC pauses when setting this value ourselves.