1 2 Previous Next 17 Replies Latest reply on Dec 30, 2007 8:34 AM by timfox

    New wireformat and client side remoting

    timfox

      To kick off the discussion, here are my initial comments on Jeff's wiki:

      In the diagram where you map MINA objects to JMS objects, we should remember that the JMS layer is a thin layer on the front.

      MINA needs to map to a JMS agnostic layer (which may look a bit like JMS).

      I'm thinking like the following:

      JMS Agnostic MINA
      --- -------- ----
      
       IoConnector IoAcceptor
       \ /
       \ IoSession/
      Connection----Connection------------------
       | |
       | |
       Session Session
       / \ \
      Producer Consumer Consumer
      

      I think Connection Session and Consumer are good abstractions for generic messaging
      But Producer and Browser only live on the JMS side.
      Producing messages and browsing can be methods on the agnostic Session class.

      IOSession will only need to maintain a map of references to the agnostic objects not the JMS objects.

      It will need to maintain such a map so the objects can be called back.

      So.. what we end up with is basically similar to the current Dispatcher but on the client side.

      In effect the system is *symettrical*. We have a dispatcher on the client side which looks up objects in order to deliver responses, and
      we have dispatcher on the server side which looks up objects to deliver requests.

      Really requests and responses are the same thing, it's just the direction of travel is different. I think this symetry is aesthetically pleasing which is often a good sign.

      Regarding the messages: Perhaps it is not such a good idea to classify the messages as synchronous or asychronous since some messages can be sent either asynchronously or synchronously - a good example is sending a message -for persistent messages they are sent synchronously, for non persistently, but the message being sent is the same.

      We should come up with a set of messages (which can be either requests or responses) and the api determines whether they are sent asynchonrous or not.

      e.g.

      interface OurRemotingAPI()
      {
      void sendOneWay(...);

      void sendBlocking(...);'

      Handler registerCallbackHandler(..);
      }

      Actually, currently the *only* messages we send asynchronously are:

      1) consumer change rate
      2) messages send (for non persistent messages)

      All the rest are synchronous.

      We could consider making the following asynchronous:

      1) Cancel delivery
      2) Start/stop connection

      In terms of how to correlate requests and responses to give synchronous calls, it should be fairly simple to do something like the following:

      1) Send message with "request id" in message.
      2) Block thread and put in map with key of request id.
      3) When response comes back, look at request id.
      4) Lookup thread in map with that request id and wake it up.

      Some kind of wait/notify should do the trick.

      Another thing to think about is management messages.

      You probably want to define a message type as a management message (or a set of management messsages).

      The idea here is the user could send a messages saying "createQueue" or "getNumberOfMeessafesInQueue" or any number of different operations that we currently do via JMX, and get results that way without JMX.

      Finally one other thing to think about is how MINA is going to support SSL and HTTP.

        • 1. Re: New wireformat and client side remoting
          timfox

          More thoughts, how this might look:

          abstract class Packet
          {
          long correlationID;
          }

          //Example
          class CreateConnectionMessage extends Packet
          {
          //all the params in here
          }


          interface Dispatcher
          {
          Handler registerHandler(int handlerID);

          void sendOneWay(Packet packet, int handlerID);

          void sendBlocking(Packet packet, int handlerID);
          }

          interface Handler
          {
          void handle(Message message);
          }

          Then you have one instance of the Dispatcher on the client side (clientDispatcher), and you have another instance of the Dispatcher on the server side (serverDispatcher)
          (Remember, it's symmetrical)

          Here are some examples of usage:

          1) Send a non persistent message to a queue on the server

          a) Create a SendMessage message
          b) Call sendOneWay on the clientDispatcher
          c) Sometime later on the server side the serverDispatcher receives the message and calls handle() on the Handler instance that was registered on the server side to handle message sends

          2) Send a persistent message to a queue on the server

          same a) b) c) as above but the
          d) server invokes sendOneWay() on the serverDispatcher with a message instance representing the response with the corelation id set.
          e) clientDispatcher receives the message and looks up correlation id in its internal map, unblocks thread with result.

          3) Send a message from server to consumer on client

          a) [The consumer will already have registered a handler with the client Dispatcher]
          b) Server invokes sendOneWay() on the serverDispatcher with the message instance.
          c) The clientDispatcher receives the mesage, looks up the appropriate handler and invokes handle() on it.
          d) Notice that there is no need for the clientDispatcher to have specific knowledge of the session and consumer objects - all it needs
          to know about are handlers! Nice and simple huh?

          4) Non blocking reliable send of message (this is over and above JMS)
          a) client invokes sendOneWay()on clientDispatcher with a message
          b) client carries on doing something else
          c) server receives message looks up handler and invokes handle() on it.
          d) server persists message and calls sendOneWay() on the serverDispatcher
          e) server carries on doing something else
          f) client receives message on calls handle() on response handler.
          g) Now client knows that messages was persisted, but since non blocking can get much better throughput not limited by network RTT

          • 2. Re: New wireformat and client side remoting
            trustin

             

            "timfox" wrote:
            To kick off the discussion, here are my initial comments on Jeff's wiki:

            In the diagram where you map MINA objects to JMS objects, we should remember that the JMS layer is a thin layer on the front.

            MINA needs to map to a JMS agnostic layer (which may look a bit like JMS).

            I'm thinking like the following:

            JMS Agnostic MINA
            --- -------- ----
            
             IoConnector IoAcceptor
             \ /
             \ IoSession/
            Connection----Connection------------------
             | |
             | |
             Session Session
             / \ \
            Producer Consumer Consumer
            

            I think Connection Session and Consumer are good abstractions for generic messaging
            But Producer and Browser only live on the JMS side.
            Producing messages and browsing can be methods on the agnostic Session class.


            Looks great to me.

            "timfox" wrote:
            Actually, currently the *only* messages we send asynchronously are:

            1) consumer change rate
            2) messages send (for non persistent messages)

            All the rest are synchronous.

            We could consider making the following asynchronous:

            1) Cancel delivery
            2) Start/stop connection

            In terms of how to correlate requests and responses to give synchronous calls, it should be fairly simple to do something like the following:

            1) Send message with "request id" in message.
            2) Block thread and put in map with key of request id.
            3) When response comes back, look at request id.
            4) Lookup thread in map with that request id and wake it up.

            Some kind of wait/notify should do the trick.


            MINA 2.x provides exactly what you want as a filter. Please take a look at here:

            http://svn.apache.org/viewvc/mina/trunk/core/src/main/java/org/apache/mina/filter/reqres/

            "timfox" wrote:
            Finally one other thing to think about is how MINA is going to support SSL and HTTP.


            MINA already has support for SSL including StartTLS. It's known to be very reliable. SSL is provided as a filter so just inserting a new SSL filter will enable SSL immediately. No sweat.

            MINA recently imported AsyncWeb source code into its sandbox, and it should be included into the official distribution after some polishing. MINA has an infra for implementing protocol codec and AsyncWeb provides HTTP codec on top of it. Therefore, you can build your own lightest-weight web server very easily. Please take a look at this example:

            http://svn.apache.org/viewvc/mina/sandbox/asyncweb/example/src/main/java/org/safehaus/asyncweb/example/lightweight/

            Please feel free to contact me or the MINA team whenever you need some support related with MINA including questions like 'does feature XXX supported in MINA?' Actually we got most, and want to add as much as possible harmonizing all the features with each other.

            • 3. Re: New wireformat and client side remoting
              jmesnil

               

              "timfox" wrote:
              To kick off the discussion, here are my initial comments on Jeff's wiki:

              In the diagram where you map MINA objects to JMS objects, we should remember that the JMS layer is a thin layer on the front.

              MINA needs to map to a JMS agnostic layer (which may look a bit like JMS).

              I'm thinking like the following:

              JMS Agnostic MINA
              --- -------- ----
              
               IoConnector IoAcceptor
               \ /
               \ IoSession/
              Connection----Connection------------------
               | |
               | |
               Session Session
               / \ \
              Producer Consumer Consumer
              

              I think Connection Session and Consumer are good abstractions for generic messaging
              But Producer and Browser only live on the JMS side.
              Producing messages and browsing can be methods on the agnostic Session class.


              You're right. I did not want to push too much emphasis on the client refactoring task (http://jira.jboss.org/jira/browse/JBMESSAGING-681 ) but it makes the picture clearer overall


              IOSession will only need to maintain a map of references to the agnostic objects not the JMS objects.

              It will need to maintain such a map so the objects can be called back.


              in fact, I'm redesigning that. The IoSession will only now about "agnostic handlers" objects. Each JBM object will then implement this handler interface.


              In effect the system is *symettrical*. We have a dispatcher on the client side which looks up objects in order to deliver responses, and
              we have dispatcher on the server side which looks up objects to deliver requests.

              Really requests and responses are the same thing, it's just the direction of travel is different. I think this symetry is aesthetically pleasing which is often a good sign.

              Regarding the messages: Perhaps it is not such a good idea to classify the messages as synchronous or asychronous since some messages can be sent either asynchronously or synchronously - a good example is sending a message -for persistent messages they are sent synchronously, for non persistently, but the message being sent is the same.

              We should come up with a set of messages (which can be either requests or responses) and the api determines whether they are sent asynchonrous or not.


              good point. I will clarify the terminology.


              Another thing to think about is management messages.

              You probably want to define a message type as a management message (or a set of management messsages).

              The idea here is the user could send a messages saying "createQueue" or "getNumberOfMeessafesInQueue" or any number of different operations that we currently do via JMX, and get results that way without JMX.


              I'll have a look at it and update http://jira.jboss.com/jira/browse/JBMESSAGING-91

              From what I understand of MINA, it should be quite straightforward to add a handler on the server-side which handlers all the "ManagementMessage" sent on the same connection that regular messages.
              This should make it simple to separate regular message logic from management message logic.

              • 4. Re: New wireformat and client side remoting
                jmesnil

                 

                "timfox" wrote:
                More thoughts, how this might look:

                abstract class Packet
                {
                long correlationID;
                }

                interface Dispatcher
                {
                Handler registerHandler(int handlerID);

                void sendOneWay(Packet packet, int handlerID);

                void sendBlocking(Packet packet, int handlerID);
                }

                interface Handler
                {
                void handle(Message message);
                }


                I have a simple prototype which is quite similar:

                interface Packet
                {
                 long correlationID;
                 String handlerID;
                }
                
                class CreateSessionRequest extends Packet {
                }
                class CreateSessionResponse extends Packet {
                }
                
                interface Client
                {
                 Handler registerHandler(int handlerID, Handler handler);
                 void unregisterHandler(int handlerID);
                
                 void sendOneWay(Packet packet);
                
                 Packet sendBlocking(Packet packet);
                }
                
                interface Handler
                {
                 void handle(Packet packet);
                }
                


                • to send a message and forget it: sendOneWay(message)
                • to send a packet and be notified later on: register the handler, set the handlerID on the packet and send it one way. Handler will be called back sometimes later.
                • to send a message and block until the response arrives: Packet response = sendBlocking(request)



                  Here are some examples of usage:
                  ...


                  I'm writing unit tests based on this simple API to check what we need to cover to integrate MINA into JBoss Messaging.


                • 5. Re: New wireformat and client side remoting
                  jmesnil

                   

                  "trustin" wrote:


                  MINA 2.x provides exactly what you want as a filter. Please take a look at here:

                  http://svn.apache.org/viewvc/mina/trunk/core/src/main/java/org/apache/mina/filter/reqres/



                  Cool, one thing less to reinvent on our side :)
                  I had a look at it and it is exactly the feature we need.

                  Thanks for the tip!

                  • 6. Re: New wireformat and client side remoting
                    timfox

                     

                    "trustin" wrote:

                    MINA already has support for SSL including StartTLS. It's known to be very reliable. SSL is provided as a filter so just inserting a new SSL filter will enable SSL immediately. No sweat.

                    MINA recently imported AsyncWeb source code into its sandbox, and it should be included into the official distribution after some polishing. MINA has an infra for implementing protocol codec and AsyncWeb provides HTTP codec on top of it. Therefore, you can build your own lightest-weight web server very easily. Please take a look at this example:

                    Please feel free to contact me or the MINA team whenever you need some support related with MINA including questions like 'does feature XXX supported in MINA?' Actually we got most, and want to add as much as possible harmonizing all the features with each other.


                    Sounds good. Thanks Trustin

                    • 7. Re: New wireformat and client side remoting
                      timfox

                       

                      "jmesnil" wrote:
                      "timfox" wrote:
                      More thoughts, how this might look:

                      abstract class Packet
                      {
                      long correlationID;
                      }

                      interface Dispatcher
                      {
                      Handler registerHandler(int handlerID);

                      void sendOneWay(Packet packet, int handlerID);

                      void sendBlocking(Packet packet, int handlerID);
                      }

                      interface Handler
                      {
                      void handle(Message message);
                      }


                      I have a simple prototype which is quite similar:

                      interface Packet
                      {
                       long correlationID;
                       String handlerID;
                      }
                      
                      class CreateSessionRequest extends Packet {
                      }
                      class CreateSessionResponse extends Packet {
                      }
                      
                      interface Client
                      {
                       Handler registerHandler(int handlerID, Handler handler);
                       void unregisterHandler(int handlerID);
                      
                       void sendOneWay(Packet packet);
                      
                       Packet sendBlocking(Packet packet);
                      }
                      
                      interface Handler
                      {
                       void handle(Packet packet);
                      }
                      



                      Sounds pretty much right to me :)

                      You could probably just use the current Dispatcher and change it to implement the new Dispatcher interface.

                      You could also re-use the current requests and reponses (of course some won't be appropriate any more) - these are the packets.

                      You would need to change the serverInvoker method of course to instead call back into the Dispatcher, so the Dispatcher can notify the correct handlers.

                      A couple more things to think about:

                      1) How does the user configure the new transport?
                      The user needs to be able to choose from:
                      a) socket
                      b) ssl
                      c) http
                      each will have its own set of parameters
                      some config is appropriate for the server side and some for the client. e.g. thread pool sizes and MINA threading model.

                      2) We need some kind of "heartbeat" functionality. I.e. periodically sending ping messages on the connection, so the server knows if the client is still alive, and the client knows if the connection is still alive.


                      • 8. Re: New wireformat and client side remoting
                        jmesnil

                        I've started to document the new Remote API on the wiki: http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossMessagingNewRemoteAPI

                        • 9. Re: New wireformat and client side remoting
                          timfox

                           

                          "jmesnil" wrote:
                          I've started to document the new Remote API on the wiki: http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossMessagingNewRemoteAPI


                          This looks good :)

                          Some comments:

                          Can targetID and callback id be longs (or ints?) too?

                          I'm thinking that Java guids are pretty long, and we want to keep the packets as small as possible - maybe I'm being over conservative though...

                          Do connect and disconnect really belong on the RemotDispatcher?

                          Three different ways to exchange packets - the second way (e.g. send non persistent message) - in fact, non persistent messages can be fire and forget - non blocking request / response is not supported using the JMS API but is very useful for doing stuff over and above JMS which we can expose through the generic interface.

                          responses via HTTP - is this done by keeping the http connection open and sending responses that way? I can't see it work otherwise without polling.

                          http post: i can see that encoding requests as http params is not always going to work - especially if we have to tunnel through another web server who will limit http params to a max length - can we extend the http codec to do this differently?

                          • 10. Re: New wireformat and client side remoting
                            jmesnil

                             

                            "timfox" wrote:

                            Can targetID and callback id be longs (or ints?) too?

                            I'm thinking that Java guids are pretty long, and we want to keep the packets as small as possible - maybe I'm being over conservative though...


                            I went for convenience in the API and using Strings for the IDs makes it possible to use Java 5 UUID and get rid of all the complexity we got from generating our own IDs (as planned by http://jira.jboss.org/jira/browse/JBMESSAGING-1040).

                            However you're right that it increases the size of the packet by doing so.
                            Depending on the payload of the packet, it may be negligible... or not.

                            For now, I'd keep the String ID and flag it as an area of optimization.


                            Do connect and disconnect really belong on the RemotDispatcher?


                            Not really :)
                            The RemoteDispatcher corresponds to both JBM 1.4.x Dispatcher (register/unregister) & JBoss Remoting Client (connect/disconnect/send).
                            Given the simplicity of the API, I keep the functions together in a single type but conceptually they belong to 2 different types.
                            As complexity grows, I'll likely refactor it.


                            Three different ways to exchange packets - the second way (e.g. send non persistent message) - in fact, non persistent messages can be fire and forget - non blocking request / response is not supported using the JMS API but is very useful for doing stuff over and above JMS which we can expose through the generic interface.


                            I'm not sure to follow you: we need to support blocking req/resp for some of our operations (e.g session creation), don't we?


                            responses via HTTP - is this done by keeping the http connection open and sending responses that way? I can't see it work otherwise without polling.


                            That's the way it seems to work: the connection corresponds to a Keep-Alive HTTP connection (kinda...)
                            I'm still not 100% clear on HTTP support: I'm able to use curl and firefox to send and receive on HTTP & HTTPS but ab is not working for now...


                            http post: i can see that encoding requests as http params is not always going to work - especially if we have to tunnel through another web server who will limit http params to a max length - can we extend the http codec to do this differently?


                            The decoding of HTTP post is done by AsyncWeb (a minimal HTTP server implementation on top which should land into MINA trunk) .
                            Trustin plan to get it integrated for MINA 2.0 (The POST parsing issue is tracked by http://jira.safehaus.org/browse/ASYNCWEB-24)[/url]

                            • 11. Re: New wireformat and client side remoting
                              timfox

                               

                              "jmesnil" wrote:


                              I'm not sure to follow you: we need to support blocking req/resp for some of our operations (e.g session creation), don't we?


                              Yes, persistent messages are sent blocking request / response, non persistent messages can be sent fire and forget (not response required at all).

                              But there is a third mode: non blocking request /response.

                              This is where you send the request, but you don't block. The server receives the request and processes it. The server then sends back a response. The response gets handled on the client side by a different thread to that which sent the request. These semantics aren't supported by JMS. But would be very useful for our generic API by giving very high throughput for persistent messaging sending since it wouldn't be limited by the network RTT as it is in the blocking case.

                              • 12. Re: New wireformat and client side remoting
                                jmesnil

                                 

                                "timfox" wrote:

                                Yes, persistent messages are sent blocking request / response, non persistent messages can be sent fire and forget (not response required at all).

                                But there is a third mode: non blocking request /response.

                                This is where you send the request, but you don't block. The server receives the request and processes it. The server then sends back a response. The response gets handled on the client side by a different thread to that which sent the request. These semantics aren't supported by JMS. But would be very useful for our generic API by giving very high throughput for persistent messaging sending since it wouldn't be limited by the network RTT as it is in the blocking case.


                                This 3rd case is meant to be covered by sendOneWay(AbstractPacket packet, PacketHandler callback).
                                For now, the response is handled by the same thread which sent the request but I plan to use an ExecutorService to have other threads take care of handling the packet


                                • 13. Re: New wireformat and client side remoting
                                  timfox

                                   

                                  "jmesnil" wrote:

                                  This 3rd case is meant to be covered by sendOneWay(AbstractPacket packet, PacketHandler callback).


                                  That's right. But I was just making the point that in the wiki page, you mention that this mode would be used for non persistent messages which is not right :)


                                  "# send a packet and set a PacketHandler? to be called back later on (e.g. send a non persistent message)

                                  * using sendOneWay(AbstractPacket? packet, PacketHandler? callbackHandler)


                                  Whereas the real use case is persistent non blocking messages.


                                  For now, the response is handled by the same thread which sent the request but I plan to use an ExecutorService to have other threads take care of handling the packet


                                  I'm not sure that will be necessary - we want to avoid as much context switching as possible.

                                  AFAIK Mina has two thread pools - once for the selectors and one to actually execute the requests (responses). Should be ok to just use the MINA thread surely?

                                  • 14. Re: New wireformat and client side remoting
                                    jmesnil

                                     

                                    "timfox" wrote:
                                    "jmesnil" wrote:

                                    This 3rd case is meant to be covered by sendOneWay(AbstractPacket packet, PacketHandler callback).


                                    That's right. But I was just making the point that in the wiki page, you mention that this mode would be used for non persistent messages which is not right :)


                                    Ok, got it! ;) I'll fix that.


                                    For now, the response is handled by the same thread which sent the request but I plan to use an ExecutorService to have other threads take care of handling the packet


                                    I'm not sure that will be necessary - we want to avoid as much context switching as possible.

                                    AFAIK Mina has two thread pools - once for the selectors and one to actually execute the requests (responses). Should be ok to just use the MINA thread surely?

                                    1 2 Previous Next