13 Replies Latest reply on Dec 7, 2006 8:40 AM by Stefano Maestri

    ws-eventing clustering problem

    Stefano Maestri Expert

      As Heiko asked me some times ago I tried to describe in deep when ws-eventing SubscriptionManager need to be clustered in our opinion. I wrote a post on our blog describing the problem and our proposed solution including a Sequence diagram to help me in the description.
      I reported here the text of the post, but I kindly suggest to read the post on the blog, because the Sequence Diagram is very useful IMHO.
      Let me know if you prefer me to transfer it to a wiki page. The post is this one:
      http://www.javalinux.it/blogs/index.php?title=ws_eventing_clustering_problem&more=1&c=1&tb=1&pb=1

      Of course we would have fun to work on this issue.


      * Subscription on one of the clustered environment's machine

      DESCRIPTION: when a client subscribes to our ws-eventing service the load balancer sends his requests to a real server that receives the subscription and store it locally (in current jbossws eventing implementation this is achieved using a Map owned by SubscriptionManagerMbean). An event source generates a notification and delegates its notification to jbossws; but in a cluster environment this event source could use a different real server knowing nothing about the subscriber. The notification would be lost. See messages from 1 to 5 in the above diagram. Of course a very similar situation may happen when renewing a subscription: a renew message cold be send to a server different from the one that received the first subscription, resulting in subscription renew failure. The client will receive an "Unable to renew" exception, and on the server side the subscription will expire for timeout! All notifications are lost until client re-subscribe to the right real server. See messages from 6 to 13 in the above diagram.

      PROPOSED SOLUTION: These two problems could be solved clustering the SubscriptionMap. The easiest solution would probably be arranged using HAJNDI to store subscriptions. Of course the use of a local Map instead of HAJNDI have to be configured at deploy time.

      * Shutdown of one of the cluster's machine

      DESCRIPTION: SubscriptionManagerMbean sends notifications to remove every subscriptions when it is stopped or the real server it runs on shutdowns. But in a cluster environment other servers continue to work and potentially to send notifications. Those notifications will be lost. See messages from 14 to 26 in the above diagram.

      PROPOSED SOLUTION: SubscriptionManagerMbeans have to be clustered and have to exchange their status before they can decide to send this kind of notifications. The solution could be implemented using jGroups (of course), but it needs further investigations and discussions, mainly to understand if SubscriptionManager is the only part of jbossws needing clustered solutions



        • 1. Re: ws-eventing clustering problem
          Heiko Braun Master

          Thanks for the detailed explanation. Now i clearly see the problem.

          The eventing specs allow for SubscriptionManager interposition. This means you can have multiple event sources being handled by just one subscription manager. In composition with WS-Addressing a subscription response contains the subscription manager EPR (endpoint reference). Which basically is the URL plus reference parameters.

          Applying this idea to your cluster topology means that you would have a single SubscriptionManager instance running on a particular node, addressed through WS-Addressing.

          Still this lacks some features that would need to be implemented:

          - SubscriptionManager runs HA-Singleton
          - There would only be one EventDispatcher, available through HA-JNDI
          - Event sources need to know the subscription manager EPR details. This could be propagated through HA-JNDI.

          Does this make sense to you?

          • 2. Re: ws-eventing clustering problem
            Stefano Maestri Expert

            Yes, it makes sense for me., it seems a good solution for cluster environment where all real server is accessible directly. In our environment (and I think is a widely used solution) we have a load balancer router in front of our real server. Clients call virtual address/port and Linux Virtual Server NAT the tcp connection to the less used real server. I think in this situation isn't possible to use your ws-Addressing proposed solution because real server can't be reached directly by client's http connection.
            What I was proposing is a distributed multiple SubscriptionManager and EventDispatcher synchronizing with jGroups. What's wrong in this solution?

            I hope I was sufficient clear, let me know if my mind seems a farm pond full of frogs :)

            • 3. Re: ws-eventing clustering problem
              Heiko Braun Master

              Your approach is perfectly valid. I just think that the problem is not directly related to WS-Eventing. It's more an addressing problem.

              Two simple use cases:

              1.) client send's subscribe request
              1.1 ) loadbalancer points it to node A
              1.3) event source on node A addresses the subscription manager (node B) HA-Singleton through HA-JNDI
              1.4) event source sends SM EPR (node B) to client

              2.) client send's renew
              2.1) loadbalancer points it to node C
              2.3) EXCEPTION: no subscription manager on node C

              OK. So far you are right.
              The load balancer doesn't know about the addressing porperties.
              But JBossWS does. Currently we can dispatch beased on WS-Addressing properties. I think this should be extended to fully support virtual addresses within a cluster topology.

              IMO this problem is not related to WS-Eventing in particular.
              It's a WS-Addressing problem that occurs in any cluster toplogy and should addressed there.

              Therefore i'd favor a common solution that's build around WS-Addressing and thus supports any WS-* extension which is composed with WSA.

              • 4. Re: ws-eventing clustering problem
                Jason Greene Master

                A better option would be to synchronize state using JBossCache, then all of the hard work is done for you. Even though you can run JBossCache in local mode, I think having some kind of abstracted state store that supports in-memory and cache would be the way to go.

                -Jason

                • 5. Re: ws-eventing clustering problem
                  Stefano Maestri Expert

                   

                  "heiko.braun@jboss.com" wrote:

                  OK. So far you are right.
                  The load balancer doesn't know about the addressing porperties.
                  But JBossWS does. Currently we can dispatch beased on WS-Addressing properties. I think this should be extended to fully support virtual addresses within a cluster topology.

                  Ok, right. I have to read something more on ws-addressing

                  "heiko.braun@jboss.com" wrote:

                  IMO this problem is not related to WS-Eventing in particular.
                  It's a WS-Addressing problem that occurs in any cluster toplogy and should addressed there.

                  100% agree

                  "heiko.braun@jboss.com" wrote:

                  Therefore i'd favor a common solution that's build around WS-Addressing and thus supports any WS-* extension which is composed with WSA.

                  Of course it's the best approach...I'm going to have a deeper look to WS specs.
                  Just one more question: it seems becoming something more widespread modification. Can I go on, or do you prefer a committer working on that?
                  And if yes, is it better it will go in jbossws 2.0?

                  • 6. Re: ws-eventing clustering problem
                    Stefano Maestri Expert

                     

                    "jason.greene@jboss.com" wrote:
                    A better option would be to synchronize state using JBossCache, then all of the hard work is done for you. Even though you
                    -Jason

                    Is this suggestion referred to my proposal or Heiko's one?

                    • 7. Re: ws-eventing clustering problem
                      Heiko Braun Master

                      I just talked to jason. He made up my mind: the addressing issue and state replication are actually two different problems.

                      In order to get you started i suggest you forget about the addressing problems for now and take a look at jboss cache for replicating the subscriptions across cluster nodes. We'd need an abstraction though, that can be addressed from the subscription manager and comes in two flavors: cluster aware (jbosscache) and in-memory (the default).
                      IMO this should be some general purpose StateManager that offers a hashmap like API to get and set data.

                      This would only leave us with the question how to deal with node shutdowns and subscription end messages...

                      • 8. Re: ws-eventing clustering problem
                        Heiko Braun Master

                         

                        Can I go on, or do you prefer a committer working on that?
                        


                        Please go on. I'd be more then happy to see more contributions like this.

                        • 9. Re: ws-eventing clustering problem
                          Heiko Braun Master

                           


                          is it better it will go in jbossws 2.0?


                          yes, working on trunk would be better. but since you are going to put in production it's your decision. If we stick with branch 1.x then it needs to merged with trunk along the way...

                          • 10. Re: ws-eventing clustering problem
                            Alessio Soldano Newbie

                             

                            "heiko.braun@jboss.com" wrote:
                            I just talked to jason. He made up my mind: the addressing issue and state replication are actually two different problems.

                            In order to get you started i suggest you forget about the addressing problems for now and take a look at jboss cache for replicating the subscriptions across cluster nodes. We'd need an abstraction though, that can be addressed from the subscription manager and comes in two flavors: cluster aware (jbosscache) and in-memory (the default).
                            IMO this should be some general purpose StateManager that offers a hashmap like API to get and set data.


                            OK, I think this is equivalent to our first proposed solution, so I think we'll go this way


                            This would only leave us with the question how to deal with node shutdowns and subscription end messages...


                            Well, ideally we would expect shutdown messages to be sent by a cluster node going down when no other node of the cluster is up, otherwise the system shouldn't generate any shutdown message. I imagine the proposed StateManager could help us in this.

                            Please go on. I'd be more then happy to see more contributions like this.

                            Perfect :-) we'll go on then; please let us know something about svn priviledges you spoke us about by email (and eventually how you manage branches)

                            yes, working on trunk would be better. but since you are going to put in production it's your decision. If we stick with branch 1.x then it needs to merged with trunk along the way...

                            Actually I think this feature would be very useful for us in production as soon as it's ready... but I fear the merge could be not so trivial... anyway I'll speak with Stefano (maeste) tomorrow and we'll let you know for sure.

                            Alessio Soldano
                            http://www.javalinux.it/blogs/

                            • 11. Re: ws-eventing clustering problem
                              Alessio Soldano Newbie

                               

                              "palin" wrote:
                              "heiko.braun@jboss.com" wrote:
                              Please go on. I'd be more then happy to see more contributions like this.

                              Perfect :-) we'll go on then; please let us know something about svn priviledges you spoke us about by email (and eventually how you manage branches)


                              OK, I've just read your last email about this, we'll create a patch for this issue and send it to you. Thank you.

                              Alessio Soldano
                              http://www.javalinux.it/blogs/

                              • 12. Re: ws-eventing clustering problem
                                Heiko Braun Master

                                Hi, i created a JIRA for this feature. Could you assign this to you?

                                http://jira.jboss.org/jira/browse/JBWS-1412

                                • 13. Re: ws-eventing clustering problem
                                  Stefano Maestri Expert

                                  I think we can't. We haven't privileges on Jira to assign issues.
                                  Of course we would like you assign it to us (use either my account "maeste" on jira, or Alessio's account "palin", it's the same for us).

                                  Do you receive my answer in private E-mail?

                                  Thanks