1 2 Previous Next 17 Replies Latest reply on Feb 17, 2005 11:59 AM by tom.elrod

    Callbacks

      I was talking with John about callbacks and we started getting into a good bit of design discussion, so wanted to post it hear for reference and any feedback.

      First off, the Callback object needs to have information about which server made the callback (for push callbacks in particular). So will include the server locator in with the Callback object for this. Will also include ability to pass a callback handle object (like is done with JMX notifications).

      Next up is pull callbacks. One issue is the storing of callbacks on the server waiting for the client to call to get the callbacks. Will need to have some server configuration to define the policy for handling memory management for these callbacks. Will likely handle in one of two ways (probably both); 1. Have some expiration on the callbacks so that if they are not picked up by a certain time, they will be thrown away and 2. persist the callbacks message depending on some size constraint (similar to way jms softens messages).

      Another idea on this is to have the remoting client (under the covers) automatically poll for callbacks at some regular interval on behalf of the user. Then when the user makes to get the callback objects, they will be returned locally (this makes callbacks for the client fairly fast). Another option is to actually piggy back callbacks on invocation responses, but only problem is can not control frequency of this (but obviously helps in performance). Maybe some combination of the two, depending on when the last invocation was made. However, this poses the memory problem now on the client side (which IMHO is better than on the server side). So will need some way to manage memory on the client side. Will also need to make this configurable, as some clients may not want to have callbacks stored in client memory space due to memory constraints.

      Ok. Fire away.

      -Tom


        • 1. Re: Callbacks
          mazz

          I really like the idea of the client under the covers periodically pulling down callbacks. This does two things:

          1) relieves the memory requirements on server from having to hold all callbacks for all clients.

          2) makes it faster for the client when it explicitly asks for the callbacks

          One thing that isn't covered in your initial post is - what happens to the status of the listener on a failure?

          For example, if I have a push callback listener defined, and for some reason a failure occurs (IOException or whatever) when trying to push a callback - does that listener immediately get removed from the list of listeners? Or, does it stay and we just log an error?

          Removing it immediately means the client will no longer get any more callbacks - if the client is dead (and that's why it failed), then this is OK and is actually what we want (no sense storing callbacks for a client that will never come get them). But if it was an intermittent problem (network went down for example, but comes back up), then the client will now no longer get any more callbacks, but it doesn't know its been de-listed so it won't know its missing callbacks.

          If you don't remove it from the list of listeners, then the opposite reasoning is valid. If a network problem caused the failure, then not de-listing the listener was the right thing to do; but if the client has gone away, each time that callback is triggered, we are going to get an error, ad-infinitum.

          We need a way to periodically clean up the list of listeners - verifying their validity and removing those that have gone away.

          Using Subscriptions would be one way to do it - but that adds more things for the client to do - it must maintain its subscriptions (a client can have multiple subscriptions to multiple servers, too). The client will have to resubscribe to its servers when its subscriptions are about to expire.

          On the server side, it must periodically reap out-dated listeners whose clients are now gone.

          Another way to do it is to implement some sort of hearbeat mechanism - periodically, the server should take the pulse of all its listeners - if it fails to see a live client on the other end, it de-lists it.

          • 2. Re: Callbacks
            mazz

            Its even worse for pull callbacks because you don't have a live endpoint that the server can "ping". So, how do you know if a client is ever going to come and pick up its callbacks? Callback listener expirations are the only way I see it can be done. You need a way to clean up dead pull listeners.

            This is different than callback expirations. Each callback can have an expiration (if its not picked up in X minutes, delete it). A callback listener should also have an expiration - how the semantics are defined is up in the air - do you have a hard expiration (kill the listener after X minutes since the listener was added) or soft (kill the listener if I haven't heard from the listener in X minutes; resetting the timer each time I've communicated with it)

            --John Mazz

            • 3. Re: Callbacks

               

              "mazz@jboss.com" wrote:
              For example, if I have a push callback listener defined, and for some reason a failure occurs (IOException or whatever) when trying to push a callback - does that listener immediately get removed from the list of listeners? Or, does it stay and we just log an error?


              An exception gets thrown back to the handler that initiated the callback on the server side. Will look like:

              org.jboss.remoting.HandleCallbackException: Error dispatching callback to handler.
              at org.jboss.remoting.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:165)
              at org.jboss.samples.callback.CallbackServer$SampleInvocationHandler.invoke(CallbackServer.java:111)
              at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:293)
              at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:169)
              at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:255)
              at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:317)
              at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:147)
              Caused by: org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection.
              at org.jboss.remoting.transport.socket.SocketClientInvoker.transport(SocketClientInvoker.java:230)
              at org.jboss.remoting.RemoteClientInvoker.invoke(RemoteClientInvoker.java:95)
              at org.jboss.remoting.Client.invoke(Client.java:197)
              at org.jboss.remoting.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:153)

              However, the client does not get removed as a callback listener on the server side. Not sure how to handle this. Maybe have the handler manually remove the callback listener from the server invoker if they want?

              The other option is for a user to include use of detectors. The detectors perform this heartbeat and will discover that a sever is dead and remove it from the list of servers in the NetworkRegistry, which can listen for in regards to callback servers and remove them as well from the ServerInvoker and handlers (of course this last part is not wired up currently).

              Detectors is a more reliable and better performing way to do this IMHO as it borrows both from subscription and heartbeat. It will monitor detections to ensure that a server's detection message does not go stale (the staleness duration is configurable). If it does, then as a last effort, will try to pint it. If this does not work, it is considered dead and removed from the NetworkRegistry.



              • 4. Re: Callbacks

                Pull callbacks are a whole different game as even detectors have no visibility here. Subscriptions may the only way to go in this case. Will have to think on it some more after 1.0.1 final release.

                • 5. Re: Callbacks
                  mazz

                  One thing came up as part of the editorial process of getting my remoting article published.

                  There may be situations (due to firewall or NAT issues) that cause the remote server to not be able to reach or send data to the client's callback server (the server that accepts incoming push callbacks).

                  Need to be able to determine when this is the case so the remote server can immediately unregister that remote callback server since it will never work.

                  Of course, I don't know how you'd do that off hand, but its something to keep in mind.

                  • 6. Re: Callbacks

                    I don't really see this as an issue remoting should address any differently. The reason is that I see this as the responsibility of the client registering the callback server. If the server is not going to be able to call back directly to the client's callback server, they should use pull callbacks.

                    If the client does register a callback server that the target server can not call back on, it will throw the exception back to the handler. I think it is important for the invocation handler to know that the callback did not succeed in this case.

                    However, would probably be good to provide an easier way for the invocation handler to remove the callback handler from the ServerInvoker (can currently do this, but takes some effort).

                    • 7. Re: Callbacks
                      starksm64

                      The issue here is one of cleanup. Its not that there is some magic at the lowest level that can transparently switch from a push to pull callback scenario on detection of a misconfigured client. A misconfigured client should not result in resource leaks and excessive errors on the server side however. A misconfigured client should be logged as a problem and then cleaned up.

                      Transient failures need to be handled by higher level connection exception listeners.

                      • 8. Re: Callbacks

                        Not sure what you are trying to say here. The remoting server will detect that the callback client is unavailable and throw an exception to the invocation handler. Then either the remoting server can remove the callback client automatically or let the invocation handler do it.

                        The only problem with having remoting do this automatically is how to figure out exactly when this should be triggered? It is possible that the callback client is only temporarily unavailable, thus removing it automatically on the first failure might be too severe. I think it should be up to the user (invocation handler) to decide how tolerant of failures they wish to be.

                        Guess could have as worst case that if get three errors, will then throw something like CallbackHandlerUnavailable exception and remove it automatically (in case invocation handler forgets to remove the callback handler themselves).

                        • 9. Re: Callbacks
                          starksm64

                          I'm saying the default behavior at the lowest level is to treat the failing client as bad and smack it. Misbehaving connections should not be resulting in wasted threads, other resource leaks, and excessive error messages.

                          • 10. Re: Callbacks

                            If the flow of the exception is:

                            client -> invocation handler -> core remoting

                            The invocation handler can choose to filter what exceptions are handled
                            by the core system and/or handle it itself.

                            Any exception that does make it back to the core system should be assumed
                            to be a failure and the client removed.

                            NOTE: The application (invocation handler?) that has exposed the remote object
                            will still need to know about the removal. e.g. JMS will need to delete
                            temporary queues/topics and non durable topic subscriptions.

                            You might want to introduce a generic handler that understands the transport
                            and what is recoverable.
                            This can work across all invocation handlers, e.g.

                            client -> transport specific error handler - > invocation handler -> core remoting

                            Is this a good enough translation Tom?

                            • 11. Re: Callbacks

                              Well, uhm, not really...

                              The call stack is:

                              invocation handler::handleCallback() -> server callback handler (internal to remoting) -> remoting callback client -> network

                              So the exception will be bubble up through the server callback handler before the invocation handler (user code).

                              I will just make so if this happens, the server callback handler will remove itself from the server invoker and the invocation handler and that will be the end of it. This way, everyone on the server side will be aware that it has been removed. Only problem is that the client will not be aware that it has been removed in the case that there was a temporary network outage on the client side.


                              • 12. Re: Callbacks

                                Why is the client unaware?
                                It needs to be aware in systems like JMS so it can invoke the ExceptionListener.

                                That is the purpose of things like pings to detect that a transport is still usable.
                                In some transports like TCP it will receive an RST when the server closes the connection:

                                java.net.SocketExcepton: Connection reset by peer
                                ....
                                


                                If the server has just become unreachable because of network topology changes
                                then you need pings and/or things like TCP read timeout to detect inactivity.

                                • 13. Re: Callbacks

                                  If I can't reach the callback server to give it a callback, how would I expect to be able to tell it that I removed it as a callback listener because I couldn't reach it?

                                  However, for detection of dead (or new) servers, this is done using discovery (detectors). This does do a ping, but is independant of core remoting, which supports callbacks (detectors run as seperate service). The two (core remoting and detectors) can be used together to accomplish what you want for JMS.

                                  • 14. Re: Callbacks

                                    Posted doc on callback store implementation at http://www.jboss.org/wiki/Wiki.jsp?page=Remoting_Callback_Configuration.

                                    1 2 Previous Next