I really like the idea of the client under the covers periodically pulling down callbacks. This does two things:

1) relieves the memory requirements on server from having to hold all callbacks for all clients.

2) makes it faster for the client when it explicitly asks for the callbacks

One thing that isn't covered in your initial post is - what happens to the status of the listener on a failure?

For example, if I have a push callback listener defined, and for some reason a failure occurs (IOException or whatever) when trying to push a callback - does that listener immediately get removed from the list of listeners? Or, does it stay and we just log an error?

Removing it immediately means the client will no longer get any more callbacks - if the client is dead (and that's why it failed), then this is OK and is actually what we want (no sense storing callbacks for a client that will never come get them). But if it was an intermittent problem (network went down for example, but comes back up), then the client will now no longer get any more callbacks, but it doesn't know its been de-listed so it won't know its missing callbacks.

If you don't remove it from the list of listeners, then the opposite reasoning is valid. If a network problem caused the failure, then not de-listing the listener was the right thing to do; but if the client has gone away, each time that callback is triggered, we are going to get an error, ad-infinitum.

We need a way to periodically clean up the list of listeners - verifying their validity and removing those that have gone away.

Using Subscriptions would be one way to do it - but that adds more things for the client to do - it must maintain its subscriptions (a client can have multiple subscriptions to multiple servers, too). The client will have to resubscribe to its servers when its subscriptions are about to expire.

On the server side, it must periodically reap out-dated listeners whose clients are now gone.

Another way to do it is to implement some sort of hearbeat mechanism - periodically, the server should take the pulse of all its listeners - if it fails to see a live client on the other end, it de-lists it.

Its even worse for pull callbacks because you don't have a live endpoint that the server can "ping". So, how do you know if a client is ever going to come and pick up its callbacks? Callback listener expirations are the only way I see it can be done. You need a way to clean up dead pull listeners.

This is different than callback expirations. Each callback can have an expiration (if its not picked up in X minutes, delete it). A callback listener should also have an expiration - how the semantics are defined is up in the air - do you have a hard expiration (kill the listener after X minutes since the listener was added) or soft (kill the listener if I haven't heard from the listener in X minutes; resetting the timer each time I've communicated with it)

--John Mazz

One thing came up as part of the editorial process of getting my remoting article published.

There may be situations (due to firewall or NAT issues) that cause the remote server to not be able to reach or send data to the client's callback server (the server that accepts incoming push callbacks).

Need to be able to determine when this is the case so the remote server can immediately unregister that remote callback server since it will never work.

Of course, I don't know how you'd do that off hand, but its something to keep in mind.

The issue here is one of cleanup. Its not that there is some magic at the lowest level that can transparently switch from a push to pull callback scenario on detection of a misconfigured client. A misconfigured client should not result in resource leaks and excessive errors on the server side however. A misconfigured client should be logged as a problem and then cleaned up.

Transient failures need to be handled by higher level connection exception listeners.

I'm saying the default behavior at the lowest level is to treat the failing client as bad and smack it. Misbehaving connections should not be resulting in wasted threads, other resource leaks, and excessive error messages.

If the flow of the exception is:

client -> invocation handler -> core remoting

The invocation handler can choose to filter what exceptions are handled
by the core system and/or handle it itself.

Any exception that does make it back to the core system should be assumed
to be a failure and the client removed.

NOTE: The application (invocation handler?) that has exposed the remote object
will still need to know about the removal. e.g. JMS will need to delete
temporary queues/topics and non durable topic subscriptions.

You might want to introduce a generic handler that understands the transport
and what is recoverable.
This can work across all invocation handlers, e.g.

client -> transport specific error handler - > invocation handler -> core remoting

Is this a good enough translation Tom?

Why is the client unaware?
It needs to be aware in systems like JMS so it can invoke the ExceptionListener.

That is the purpose of things like pings to detect that a transport is still usable.
In some transports like TCP it will receive an RST when the server closes the connection:

java.net.SocketExcepton: Connection reset by peer
....