Potential problem with thread usage in pingers and lease

[timfox]

Looking at the code - I can see that connection pinger is implemented by creating a new java.util.Timer instance per Client on the client side, and using another java.util.Timer instance per connection on the server side.

Each java.util.Timer instance has it's own thread.

I'm worried that this is not going to scale.

Thousands of connections on the server will mean thousands of threads.

Same on the client side.

I know JBossMQ has issues with too many threads and I'd really like to avoid making the same mistakes in messaging if at all possible.

How hard would it be to change it so the same Timer (or a limited number of Timers) were used to co-ordinate the pinging and server lease functionality for multiple connections?

[starksm64]

Right. Look at using the org.jboss.util.TimerQueue, and this should be an injectable element that could be picked up from a kernel/server level instance.

[timfox]

Added JIRA task:

http://jira.jboss.com/jira/browse/JBREM-342

[tom.elrod]

Done

[timfox]

If all the pingers for a single client are using the same TimerQueue, the ping requests will be sent in sequence, right?

If the client has 100 connections open, and needs to ping each one. And each ping invocation takes 500 ms (for the sake of argument), then we can only ever produce a max. of 1 ping per 50 seconds for each connection.

So the max ping period is going to vary as A + Bn where A and B are constants and n is the number of connections. (something like this).

Only way I can see around this is to make sure the client ping period is configurable separately from the lease period and in general make the ping period much less than the lease period.

[starksm64]

Any shared timer thread (be it a single thread or a pool) needs to have the timer dispatch managed so that notifications are not lost due to an inability to keep up with expirations. The timer event should just be marking a connection as needing a ping/pong event rather than actually sending it.

For massive connection counts, we will may have to have a notion of coalescing ping events into windows of time with a timer event marking sets of connections as ready.

The timer layer probably needs a notion of not keeping up with expirations. This is something like if the next computed expiration is less than or equal to 0 for some extended period of time, the timer layer does not have enough threads.

[tom.elrod]

Have added behavior so there is now a lease window in which the ping must be received within. Initially, this window is the same value as the lease period. If the time it takes for the ping to come in (calling this the ping duration) is longer than half the ping window, the window will be increased to twice that of the ping duration. See jira issue for more details.

[dlofthouse]

I am currently looking at a related issue to this in how timers are used within the app server.

http://jboss.org/index.html?module=bb&op=viewtopic&t=80483

I have looked at the remoting code and I can see remoting did switch to using the 'TimerQueue' but at the point it was converted to a singleton you have switched back to 'java.util.Timer' was this switch intended?

[tom.elrod]

Tim found that the behavior of the TimerQueue was not going to work for what we wanted. IIRC, the issue was that if a task was rescheduled, the original task was not getting removed.

[starksm64]

So we are back to unbounded threads for handling timeouts?

[tom.elrod]

No. The java.util.Timer used for leasing is basically a singleton. For every client lease that is established, a TimerTask will be scheduled with the same Timer instance. The Timer uses a single thread for executing the TimerTasks. So will only be one thread used regardless of the number of clients.