10 Replies Latest reply on Sep 20, 2007 10:03 AM by richard_opalka

    WS-RM Sender/Receiver Location Within JBossWS Stack

    thomas.diesler

      Richard sais:

      RM SENDER AND RECEIVER LOCATION WITHIN JBOSSWS STACK

      RM Sender and Receiver must operate on top of the transport layer
      so they are able to create and close new transport connections.

      RM Sender and Receiver are RM entities which goal is to ensure
      message delivery and reliability according to the specified RM policy.

      The expected model of RM sender/receiver is:
      * It will be a separate thread responsible for accepting/returning
      messages from/to JBossWS messaging layer
      * It will persist all messages going through it
      * It will query messages in the RM store against specific conditions
      * It will create RM tasks and put it to the RM task queue when re/send is necessary

      Picture of the client side follows (RM RECEIVER is similar):

      
       +-----------------------------------------------------------------------------------+
       | RM CHANNEL |
      +---+ | |
      | | | +-----------+ +---------+ |
      | J | send(request) | | put(RM request) | | |
      | B |------------------->| |------------------->| RM | |
      | O | | | RM SENDER | | TASKS | |
      | S | return response | | return RM response | QUEUE | |
      | S |<-------------------| |<---------+ | | |
      | W | | +-----------+ | +---------+ |
      | S | | | A | | |
      | | | | | +----------+ | |
      | M | | | | | | |
      | E | | write(request) | query | read(request) | | |
      | S | | write(response) | | read(resposne) | | take RM task |
      | S | | | | | | |
      | A | | V | | | |
      | G | | +-----------+ +---|------|------------------------+ |
      | I | | | | | | | RM THREAD POOL | |
      | N | | | RM STORE | | | V | |
      | G | | | | | +----------+ +----------+ | |
      | | | +-----------+ | | Thread 1 | ... | Thread N | | |
      | L | | | +----------+ +----------+ | |
      | A | | | A | | |
      | Y | | +---|------|------------------------+ |
      | E | | | | |
      | R | | read(response)| | write(request) |
      | | | | | |
      +---+ | +----|------|------------------------+ |
       | | | V | | +---+
       | | | +-----------------------+ | | | |
       | | | | Output Stream |-------->| N |
       | | | +-----------------------+ | | | E |
       | | | | | | T |
       | | | REMOTING TRANSPORT | | | W |
       | | | | | | O |
       | | +---------------------+ | | | R |
       | | | Input Stream |<-----------------| K |
       | | +---------------------+ | | | |
       | | | | +---+
       | +------------------------------------+ |
       | |
       +-----------------------------------------------------------------------------------+
      
      


      Definitions:

      NETWORK - Intranet or Internet network

      JBOSSWS MESSAGING LAYER - JBossWS messaging layer

      RM CHANNEL - is responsible for message acceptance from the JBOSSWS MESSAGING LAYER.
      Since this time this channel is responsible for reliable message delivery
      and will comunicate with JBOSSWS MESSAGING LAYER accordingly
      (i.e. ensuring message in order delivery etc ...)

      RM STORE - is responsible for persistent operations. It must be able to:
      * write message and its associated metadata,
      * read the message and its associated metadata,
      * query the messages in the store according to the specified criteria
      NOTE 1: Messages and their metadata must be fully de/serializable without
      dependency on the context class loader. That means all messages will be
      de/serialized before putting them to the RM STORE.
      NOTE 2: Messages and their metadata will be de/serialized in the same
      thread that initiated the invocation because of the heavy use
      of ThreadLocals in the JBossWS invocation framework. (However this
      requirement could be fixed by removing the ThreadLocal dependency)

      RM TASKS QUEUE - holds RM tasks identified by the RM SENDER/RECEIVER that have to
      be send to the transport layer

      RM THREAD POOL - holds RM threads that are responsible for doing the tasks stored
      in the RM TASKS QUEUE and execute them against REMOTING TRANSPORT

      NOTE:

      All this is about messages acceptance from the JBossWS messaging layer and ensuring the reliable
      delivery to the network and than back to the mesaging layer. It is not about the RM message
      creation on the SOAP layer. SOAP headers related to the WS-RM specification will be handled
      on higher level.

      IMPLEMENTATION LOCATION:

      I decided that the most suitable place to put RM CHANNEL to is the RemotingConnectionImpl class.

        • 1. Re: WS-RM Sender/Receiver Location Within JBossWS Stack
          heiko.braun

          Nice ascii art.

          * Does a RM CHANNEL model the sender as well as the receiver side?
          * "I decided that the most suitable place to put RM CHANNEL to is the RemotingConnectionImpl class."

          Sorry, i dont understand this part.

          • 2. Re: WS-RM Sender/Receiver Location Within JBossWS Stack
            richard_opalka

            1.) Yes, the model of the SENDER and RECEIVER should be similar. I've just draw the client side, but you can imagine similar mechanism on the server side too. Just for clear in present I'm mainly focusing on the client side.
            2.) I was playing games and doing some experiments with JBossWS client side invocation framework and was looking for the most suitable place to put the RM SENDER in. I have implemented dummy sender and executed all the JBossWS tests on all JBoss AS version against this dummy sender. The result of my experiment is in SVN repository (in my branch).
            You can find this dummy sender implementation in RemotingConnectionImpl.java.

            If you would like to explain something we can communicate it also e.g. via Skype.

            I'm expecting another questions, comments, criticism and opinions from you guys ;-)

            Richard

            • 3. Re: WS-RM Sender/Receiver Location Within JBossWS Stack

              Hi Richard,
              I've just taken a look at both your private branch code and this post ;-)

              "richard_opalka" wrote:
              1.) Yes, the model of the SENDER and RECEIVER should be similar. I've just draw the client side, but you can imagine similar mechanism on the server side too. Just for clear in present I'm mainly focusing on the client side.

              OK, as a matter of fact speaking of RM, client side and server side do a lot of things the same way.

              2.) I was playing games and doing some experiments with JBossWS client side invocation framework and was looking for the most suitable place to put the RM SENDER in. I have implemented dummy sender and executed all the JBossWS tests on all JBoss AS version against this dummy sender. The result of my experiment is in SVN repository (in my branch).
              You can find this dummy sender implementation in RemotingConnectionImpl.java.


              But there's something that is not clear to me: let's consider the server side (but this also apply on client side), shouldn't the RM subsystem process incoming messages before passing them to the core? (for example to be able to send receive acks) As far as I understand, when you do all the RM processing in the RemotingConnectionImpl, all the core/application work has already been performed (you're ready to send response message, right?)

              When working on ws-policy, we thought a bit about ws-rm too, since this is going to use policies. We immagined that we could have ws-rm handlers (set by the ws-policy deployer on both client and server sides) intercepting messages and passing them to RM subsystem. May be this is not the best idea... but you might consider this too ;-)

              I'm expecting another questions, comments, criticism and opinions from you guys ;-)


              Just an idea for now... this afternoon I was thinking about cluster with Stefano. Did you consider that a real reliable messaging system is most likely going to be deployed on a clustered environment? (I mean, if you're seeking for reliability... why having a single point of failure using a single host?) IMHO it might be a plus to design the RM implementation keeping this cluster-issue in mind.

              Bye
              Alessio Soldano

              • 4. Re: WS-RM Sender/Receiver Location Within JBossWS Stack
                heiko.braun

                Maybe i need to look at the code. But i was thinking similiar. My first impression was, that RM uses a cleanly seperated subsystem, with jbossws dispatching to it (or registering a callback on the server side). Let the dispatch point be handler or whatever fits into the picture.

                Could it be that you require fine grained control over the transport?
                I.e. dealing with return codes and continuation? Is that the reason you chose the RemotingConnectionImpl.java?

                Another drawback is that the RemotingConnection is only available on the client side. How is the server side dealing with it?

                • 5. Re: WS-RM Sender/Receiver Location Within JBossWS Stack
                  richard_opalka

                   

                  "palin" wrote:

                  Just an idea for now... this afternoon I was thinking about cluster with Stefano. Did you consider that a real reliable messaging system is most likely going to be deployed on a clustered environment? (I mean, if you're seeking for reliability... why having a single point of failure using a single host?) IMHO it might be a plus to design the RM implementation keeping this cluster-issue in mind.


                  There's a lot of issues related to the Cluster vs. Web Services.
                  First of all I need to say that clustered environment is the most suitable for stateless web services. It's because if you have stateless services, you do not need to replicate sessions and cluster manager can forward message to any cluster node of his choice.
                  The situation becomes complicated if your web service isn't stateless (and each RM enabled endpoint isn't stateless). Imagine the situation you have RM receivers running on multiple cluster nodes. What you can do in such situation? In real life there is the loadbalancer in the front of cluster and if you have the good one it is cookies aware.
                  Thus the solution for clustered RM environment is:
                  * use cookies aware loadbalancer in the front of cluster
                  * all your RM endpoints (hosted on cluster nodes) must be cookies aware
                  * your cluster must be able to replicate the sessions and cookies
                  * use clustered database (where RM messages are stored)
                  * your cluster can't be hidden behind the filewall
                  If you will meet all these preconditions I don't see the problem to have clustered RM ;-)

                  Richard

                  • 6. Re: WS-RM Sender/Receiver Location Within JBossWS Stack
                    richard_opalka

                    I'll do the clarification first. What is the RM subsystem for me?
                    * RM JAXWS handlers communicating with RM store
                    * RM Channel (as drawn on the picture above) communicating with RM store too (represents reliable transport wrapper for me only)
                    * RM store (database based persistent store)
                    * RM sender/receiver used by RM channel and RM handlers too

                    What is the above picture about? It's only about reliable resending of processed and accepted messages on the transport level. Alessio and Heiko ask me why I decided to put RM channel to the RemotingConnectionImpl and not to the handler?
                    My answer is: RM sender must be placed on top of transport layer because of his resending capability.
                    I can't put it to the JAXWS handler. If I would do it, the problem could be with WS-Security (next JAXWS handler that could be in the chain). If I would communicate with transport layer directly from RM handler, WS-Security handler wouldn't be called at all and thus
                    WS security wouldn't be supported in combination with WS-RM.
                    Thus what I need is to ensure that all handlers in the handler chain are executed, and then, on transport layer RM channel will accept such completely processed outgoing messages and resend them.

                    I hope it's now more clear for you now ;-)

                    Richard

                    • 7. Re: WS-RM Sender/Receiver Location Within JBossWS Stack
                      richard_opalka

                       

                      "heiko.braun@jboss.com" wrote:
                      Could it be that you require fine grained control over the transport?
                      I.e. dealing with return codes and continuation? Is that the reason you chose the RemotingConnectionImpl.java?

                      Yes Heiko, you're absolutely right. I need the full control over the transport layer.

                      "heiko.braun@jboss.com" wrote:
                      Another drawback is that the RemotingConnection is only available on the client side. How is the server side dealing with it?

                      I'm working on client side only in present. When I will code and think about the server side I will clarify my decisions. Maybe I will change my mind during the development. For now I study a lot of JBossWS code and can't tell you how I will really implement it finally.

                      Richard

                      • 8. Re: WS-RM Sender/Receiver Location Within JBossWS Stack
                        richard_opalka

                         

                        "palin" wrote:
                        But there's something that is not clear to me: let's consider the server side (but this also apply on client side), shouldn't the RM subsystem process incoming messages before passing them to the core? (for example to be able to send receive acks) As far as I understand, when you do all the RM processing in the RemotingConnectionImpl, all the core/application work has already been performed (you're ready to send response message, right?)

                        When working on ws-policy, we thought a bit about ws-rm too, since this is going to use policies. We immagined that we could have ws-rm handlers (set by the ws-policy deployer on both client and server sides) intercepting messages and passing them to RM subsystem. May be this is not the best idea... but you might consider this too ;-)


                        RM subsystem will be hooked to the whole invocation framework, it will consists of handlers, transport wrappers, database based store and other entities. It won't be implemented on the transport layer only.

                        • 9. Re: WS-RM Sender/Receiver Location Within JBossWS Stack

                          Hi Richard,
                          first of all, just to be clear, I'm not saying clustering rm is a must neither it's easy to implement it. I actually believe it would be very usefull in a real application and that it can be obtained with less pain if the whole rm subsystem it designed keeping it in mind.
                          So... just a couple of ideas, imho we should open another thread on this in future if needed.

                          * use cookies aware loadbalancer in the front of cluster
                          * all your RM endpoints (hosted on cluster nodes) must be cookies aware

                          Are cookies really needed for rm? In any case the user loadbalancer choice is not our business ;-)

                          * your cluster must be able to replicate the sessions and cookies
                          * use clustered database (where RM messages are stored)

                          For example jboss cache /pojo cache could be used to access and replicate the sessions / rm store on all cluster nodes.

                          Once again... I'm just wondering if this could be done if the whole system is designed with this issue in mind.


                          Regarind the original post subject i.e. the RMChannel location:
                          RM subsystem will be hooked to the whole invocation framework, it will consists of handlers, transport wrappers, database based store and other entities. It won't be implemented on the transport layer only.

                          OK, now I get it. I'll be glad to share ideas about this when you'll think about the server side. I'll think about the handler issue you wrote in the meantime.

                          Bye
                          Alessio Soldano

                          • 10. Re: WS-RM Sender/Receiver Location Within JBossWS Stack
                            richard_opalka

                             

                            "palin" wrote:
                            It can be obtained with less pain if the whole rm subsystem it designed keeping it in mind.

                            I fully agree with you! We have to keep it in mind.
                            "palin" wrote:

                            Are cookies really needed for rm? In any case the user loadbalancer choice is not our business ;-)

                            If you have statefull services deployed to the cluster then cookies aware cluster nodes is a must. As I know almost all hardware/software loadbalancers are cookies aware.
                            "palin" wrote:

                            I'll think about the handler issue you wrote in the meantime.

                            I'm looking forward for your ideas ;-)

                            Richard