13 Replies Latest reply on Oct 12, 2006 2:04 AM by timfox

    JBMESSAGING-519 - Failover(HA) design discussion

    clebert.suconic

      I have created a WIKI page with a diagram with my initial ideas for HA.

      I have discussed them offline with Tim, so at least I guess I'm at a right direction.

      http://wiki.jboss.org/wiki/Wiki.jsp?page=NewMessagingHADesign

      Lets use this thread to discuss any issues with HA design.


      Clebert

        • 1. Re: JBMESSAGING-519 - Failover(HA) design discussion
          timfox

          I wouldn't use JBossConnection to manage the reconnection/recreation of the objects.

          All the classes in the org.jboss.jms.client package are intended to be very thing wrappers over the AOP client stack which contains the client functionality.

          I think a better solution would be to create a new Aspect which manages all this stuff.

          • 2. Re: JBMESSAGING-519 - Failover(HA) design discussion
            clebert.suconic

            I wouldn't do it directly on JBossConnection either.
            I was just raising the relationships that I will have to take care of.

            I will use AOP, but one thing that I was wondering is if I would need to create an extension of JBossConnection or not (JBossConnectoinHA, JBossSessionaHA, and so on), and having the aspects bount to HA classes.

            • 3. Re: JBMESSAGING-519 - Failover(HA) design discussion
              timfox

              The aspects are bound to the delegate classes anyway, not the JBossXXX classes.

              But I don't think there's a need to create subclasses. I would just use the current classes.

              BTW the hierarchy of Connection/Session/Consumer etc is already maintained using the org.jboss.jms.client.state classes by the StateCreationAspect, so you could hook into that to make your life easier.

              • 4. Re: JBMESSAGING-519 - Failover(HA) design discussion
                clebert.suconic

                I was a little bit confused about the hierarchy defined on state.

                So I could produce a diagram showing how HierarchicalStateSupport would define the dependency between Connections,Sessions, Producers, etc. (look at the wiki page again with the update):

                http://wiki.jboss.org/wiki/Wiki.jsp?page=NewMessagingHADesign

                I have done some refactoring into package state, just to show these relationships in a better way.

                I will run testcases locally and commit these changes after that.

                • 5. Re: JBMESSAGING-519 - Failover(HA) design discussion
                  ovidiu.feodorov

                  The quotes that follow are from the HA Design document (http://wiki.jboss.org/wiki/Wiki.jsp?page=NewMessagingHADesign)

                  Clebert wrote:

                  The idea is to have JBossConnection knowing the current state of subscriptions, producers and listeners.


                  Could you please qualify this a little bit more? The state of consumer/producers, as well as the state of their parent session and connection is maintained by their corresponding client-side delegates, as well as their server-side endpoints. So I don't exactly understand what you are trying to say with "have JBossConnection knowing the current state of subscriptions, producers and listeners".


                  Clebert wrote:

                  JBossConnection (or an extension of that class) will have a list of other servers to be connected in case of failure.


                  How do you envision this list to be initially populated. How it is kept in synch?

                  Clebert wrote:

                  One point we are leaving opened at this point is how to recover redeliviries of eventual already received messages.


                  Are you talking about persistent or non-persistent messages here?

                  Clebert wrote:

                  JBossConnectionHA will be responsible for capturing creation events, storing then in a list and recreate its delegates when a Connection (RemotingConnection) is failed.


                  How about we don't have any JBossConnectionHA? A JBossConnection is "failover enabled" or not, depending on whether the "FailoverAspect" or "ClusterAspect" is present in its aspect stack or not.


                  • 6. Re: JBMESSAGING-519 - Failover(HA) design discussion
                    clebert.suconic

                     

                    Could you please qualify this a little bit more? The state of consumer/producers, as well as the state of their parent session and connection is maintained by their corresponding client-side delegates, as well as their server-side endpoints. So I don't exactly understand what you are trying to say with "have JBossConnection knowing the current state of subscriptions, producers and listeners".


                    When I said JBossConnection I meant somewhere on the structure of the connection. I didn' t mean to implement this state on JBossConnection. This will probably go into ConnectionState.

                    How do you envision this list to be initially populated. How it is kept in synch?


                    At this point we have the list of available server at both PostOfficers (queue and topic). AcceptView is populating the list.
                    We will however need to decide how to communicate clients when a new node arrives on the cluster, but we can decide that later.

                    How about we don't have any JBossConnectionHA? A JBossConnection is "failover enabled" or not, depending on whether the "FailoverAspect" or "ClusterAspect" is present in its aspect stack or not.



                    Sounds good. I didn't like the idea of extending connection objects anyway. Was just listing possibilities.

                    • 7. Re: JBMESSAGING-519 - Failover(HA) design discussion
                      clebert.suconic

                      I have changed the document to have (Connection related object) instead of JBossConnection.

                      • 8. Re: JBMESSAGING-519 - Failover(HA) design discussion
                        brian.stansberry

                        Regarding use of aspects for handling failover/loadbalancing, have a look at the following classes in the AS aspects module, o.j.aspects.remoting package:

                        1) ClientChooserInterceptor -- client side aspect.
                        2) ReplicantsManagerInterceptor -- server side aspect that propagates cluster topology changes back to the client as response metadata.
                        3) ClusteredRemoting. This is a class that creates proxies that uses these interceptors.

                        EJB3 uses the first two. It doesn't actually use ClusteredRemoting; rather it does similar things in its own more specialized proxy factories.

                        These classes import the main loadbalancing stuff from the AS cluster module. I've gotten a version of that load balancing stuff extracted out into a separate project that can be checked in. For background on that see
                        http://www.jboss.com/index.html?module=bb&op=viewtopic&t=89015

                        • 9. Re: JBMESSAGING-519 - Failover(HA) design discussion
                          clebert.suconic

                          I'm creating a method called failoever(ConnectionDelegate newDelegate) into ConnectionDelegate.

                          I could already have a simple producer failing over another node after the method failoever was called.


                          Looking at this testcase you will have an idea how this is supposed to work. Onc the method failoever(connection2.getDelegate()) is called the producer is now connected to the new node.

                          public void testSimpleWithOneProducer() throws Exception
                           {
                           JBossConnection conn = (JBossConnection)this.factoryServer1.createConnection();
                           Session session = conn.createSession(false,Session.AUTO_ACKNOWLEDGE);
                           Destination destination = (Destination)getCtx1().lookup("queue/testQueue");
                           MessageProducer producer = session.createProducer(destination);
                          
                           Message message = session.createTextMessage("Hello Before");
                           producer.send(message);
                          
                           ClientConnectionDelegate delegate = (ClientConnectionDelegate)conn.getDelegate();
                           ConnectionState state = (ConnectionState)delegate.getState();
                          
                           JBossConnection conn2 = (JBossConnection)this.factoryServer2.createConnection();
                           conn.getDelegate().failOver(conn2.getDelegate());
                          
                           message = session.createTextMessage("Hello After");
                           producer.send(message);
                           }
                          
                          


                          I have a question about what would happen with Producers when dealing with transactions, as when the producer is failedOver the new connection, All the ObjectIDS (referenced on the current TransactionContext) won't exist on the new node.

                          Should we throw an exception next sendMessage or commit is called?

                          • 10. Re: JBMESSAGING-519 - Failover(HA) design discussion
                            clebert.suconic

                             

                            public void testSimpleWithOneProducerTransacted() throws Exception
                             {
                             JBossConnection conn = (JBossConnection)this.factoryServer1.createConnection();
                             Session session = conn.createSession(true,Session.AUTO_ACKNOWLEDGE);
                             Destination destination = (Destination)getCtx1().lookup("queue/testQueue");
                             MessageProducer producer = session.createProducer(destination);
                            
                             Message message = session.createTextMessage("Hello Before");
                             producer.send(message);
                            
                             ClientConnectionDelegate delegate = (ClientConnectionDelegate)conn.getDelegate();
                             ConnectionState state = (ConnectionState)delegate.getState();
                            
                             JBossConnection conn2 = (JBossConnection)this.factoryServer2.createConnection();
                             conn.getDelegate().failOver(conn2.getDelegate());
                            
                             message = session.createTextMessage("Hello After");
                             producer.send(message);
                             session.commit();
                             }
                            


                            Just completing my last post, if I used the same testcase in a transacted scenario (with pending transactions) I would get this exception:

                            org.jboss.aop.NotFoundInDispatcherException: Object with oid: 47 was not found in the Dispatcher
                             at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85)
                             at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
                             at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1008)
                             at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:857)
                             at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:454)
                             at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:541)
                             at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:261)
                             at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:172)
                             at org.jboss.remoting.Client.invoke(Client.java:589)
                             at org.jboss.remoting.Client.invoke(Client.java:581)
                             at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
                             at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N4986868250254447300.invokeNext(ClientConnectionDelegate$sendTransaction_N4986868250254447300.java)
                            


                            I guess I should throw an exception if there are pending transactions and a failover happened. I think that would be the expected signature. (Anyone disagrees?)


                            • 11. Re: JBMESSAGING-519 - Failover(HA) design discussion
                              clebert.suconic

                              Never mind: Answering my own question:

                              TxState has the list of previously sent messages associated with the current transaction. I will have to resend then if a failOver event happened.

                              • 12. Re: JBMESSAGING-519 - Failover(HA) design discussion
                                clebert.suconic

                                Could fix the MessageProducer under transaction environment.

                                Now I have a testcase sending messages in a transacted session, right before calling commit I called failOver, and all the messages were received on the new node.

                                Basically I created a method called transferHAState under DelegateSupport what will set the ID (copying it from the new server's object).

                                That is allowing handleFailoever on ConnectionAspect to reconnect objects to the new server and change IDs to the expected IDs on the new server.

                                It's working nicely now.


                                Clebert

                                • 13. Re: JBMESSAGING-519 - Failover(HA) design discussion
                                  timfox

                                  What are these ID's you are referring too?

                                  Can't you just create a new producer? Or maybe I am misunderstanding what you are doing?