10 Replies Latest reply on Jan 24, 2003 2:09 PM by belaban

    Concurrency in SFSB Replication

      Hi.

      I'm beginning to write a document on JBoss Clustering for an accademic research project. I had some more questions on the SFSB Replication algorythm.
      I wanted to know if the sfsb state broadcast operation was blocking. In other words, is the reply to the client sent only after ensuring that the state update message is delivered correctly to all running nodes in the primary partition? If this was not the case a client could do a new request before the previous state update was delivered to all the nodes, thus leading to inconsistency, right?
      But this trick doesn't work in multithreaded clients. A good solution for this could be to control concurrent access from the stub of a sfsb. Is it the way it is done now?
      The stub trick will deal with RMI clients but we have a problem with HTTP based clients. Let's say a browser is accessing concurrently two servlets via a HTML page with aframeset. The two servlets are situated on different nodes and they both access the same sfsb locally. In this case we have two sfsb replicas on different nodes that are accessed concurrently. Is this case handled in some way, except from forcing sticky http sessions? If yes how?
      I apologize for raising such complicated questions before having examined the code in detail, but I need some guidelines before i can understand the code.
      Thank you for the consideration,

      Jaksa

      PS: Where can I find some info on how is merging network partitions handled in JBoss?

        • 1. Re: Concurrency in SFSB Replication
          belaban

          Hi Jaksa,

          currently, all updates are asynchronous; the client returns immediately after broadcasting the update. At this time you cannot be guaranteed that all nodes received and applied the update.

          That's the reason why we recommend sticky sessions.

          That said, Sacha has written the code in such a way that he could easily switch to making the updates synchronous: a client call that modified a session would only return after all the nodes received the update.
          We can make this behavior configurable.

          JavaGroups (which is used for broadcasting the updates to all nodes) will soon have functionality which allows you to
          - send asynchronous updates (the one currently used in JBoss Clustering)
          - send synchronous updates (with optional timeout)
          - send synchronous updates with locking


          We envisage making the replication *mode* configurable in Clustering further down the road.

          Bela

          • 2. Re: Concurrency in SFSB Replication

            Does this mean that if we use Round Robin invocation policy from an RMI client, the state could be not updated between two requests? I have noticed a change of behaviour in this situation between the version 3.0.3 and 3.0.4. Looks like in 3.0.4 the things work fine.
            Then again if we use sticky sessions, and the node we are talking to crashes, we could switch to another node and make a new request even if there is a pending state transfer between the nodes. And if that state update arrives after the request, we get an inconsistent state.
            Or maybe there is some kind of mutual exclusion between the nodes when they access the same session bean instance?
            Another question is: do the state updates originated from the same node arrive in the exact order to the other nodes?

            Jaksa

            • 3. Re: Concurrency in SFSB Replication
              belaban

              > Does this mean that if we use Round Robin invocation
              > policy from an RMI client, the state could be not
              > updated between two requests?

              Yes. Most of the time you will not notice this because the multicast between the nodes is faster then your next RMI call from client to server.


              > Then again if we use sticky sessions, and the node we
              > are talking to crashes, we could switch to another
              > node and make a new request even if there is a
              > pending state transfer between the nodes. And if that
              > state update arrives after the request, we get an
              > inconsistent state.

              Correct.


              > Or maybe there is some kind of mutual exclusion
              > between the nodes when they access the same session
              > bean instance?

              No. The locking is currently only per node, so concurrent access to the same instance on the same node is serialized. Sacha, please correct me if I'm wrong.


              > Another question is: do the state updates originated
              > from the same node arrive in the exact order to the
              > other nodes?

              Yes. We use FIFO, so a node will see all updates from a member X in the same order. However, this says nothing about ordering between members X's and Y's messages.

              E.g. X sends m1, m2 and Y sends m1, m2.

              Valid message reception at a member M could be:

              X.m1, X.m2, Y.m1, Y.m2

              or

              X.m1, X.m2, Y.m1, Y.m2

              or

              Y.m1, Y.m2, X.m1, X.m2

              or

              Y.m1, X.m1, X.m2, Y.m2

              etc


              You see ? FIFO simply says that

              X.m1 --> X.m2

              and

              Y.m1 --> Y.m2

              where --> represent "followed by"

              Bela

              • 4. Re: Concurrency in SFSB Replication
                slaboure

                Bela,

                that's too bad because I already answered to Jaska but during the Forum change, my message has been lost...

                Nevertheless, the current implementation for SFSB does *synchronous* replication (do not confuse HTTP session and SFSB)

                • 5. Re: Concurrency in SFSB Replication
                  slaboure

                  > > Then again if we use sticky sessions, and the node
                  > we
                  > > are talking to crashes, we could switch to another
                  > > node and make a new request even if there is a
                  > > pending state transfer between the nodes. And if
                  > that
                  > > state update arrives after the request, we get an
                  > > inconsistent state.
                  >
                  > Correct.

                  no, default behaviour is synchronous for SFSB, see org.jboss.ha.hasessionstate.server.HASessionStateImpl.setState()

                  I guess maybe you did something with Jules so that the timeout is set to 0 or something like that which may quickly return but by default, the timeout is not 0.

                  > > Or maybe there is some kind of mutual exclusion
                  > > between the nodes when they access the same
                  > session
                  > > bean instance?
                  >
                  > No. The locking is currently only per node, so
                  > concurrent access to the same instance on the same
                  > node is serialized. Sacha, please correct me if I'm
                  > wrong.

                  Concurrent accesses on SFSB are not allowed by the EJB spec => exception is raised. However no distributed locking scheme exist so while concurrent calls on a single node will be detected, concurrent calls on multiple nodes may not be detected (it is a best effort). When it is detected, an exception is thrown to the naughty concurrent thread.

                  • 6. Re: Concurrency in SFSB Replication

                    I have been discussing the issue of the synchronous broadcast with Bartoli, and it seems that we were having some misuderstandings on the meaning of synchronous.
                    When you say SFSB state is replicated by a synchronous broadcast do you mean:
                    1) The sender bradcasts the state to the primary partition, waits to get an acknowledgement from all the nodes and then returns the reply to the client.
                    or
                    2) The sender broadcasts the state to the primary view, waits to receive it's own message and replies to the client without knowing if all the members of the primary partition received it's state.
                    TIA,

                    Jaksa

                    • 7. Re: Concurrency in SFSB Replication
                      belaban


                      > no, default behaviour is synchronous for SFSB, see
                      > org.jboss.ha.hasessionstate.server.HASessionStateImpl.setState()


                      I should have read the code first ...


                      > Concurrent accesses on SFSB are not allowed by the
                      > EJB spec => exception is raised. However no
                      > distributed locking scheme exist so while concurrent
                      > calls on a single node will be detected, concurrent
                      > calls on multiple nodes may not be detected (it is a
                      > best effort). When it is detected, an exception is
                      > thrown to the naughty concurrent thread.


                      Note that we are working on a mechanism where all access to entity beans and stateful session beans will be through a *replicated* cache.
                      Locking all instances of a bean before a write-access will be done by the cache. We will essentially provide distributed locking.

                      Have a look at the Cache/JBoss forum if someone is interested.

                      Bela

                      • 8. Re: Concurrency in SFSB Replication
                        slaboure

                        Hello Jaksa,

                        Answer is 1).

                        Cheers,


                        sacha

                        • 9. Re: Concurrency in SFSB Replication
                          slaboure

                          > I should have read the code first ...

                          Don't worry, I had to look at it myself ;)

                          > Note that we are working on a mechanism where all
                          > access to entity beans and stateful session beans
                          > will be through a *replicated* cache.
                          > Locking all instances of a bean before a write-access
                          > will be done by the cache. We will essentially
                          > provide distributed locking.

                          yes, that will be very cool and will simply the SFSB code a lot. Plus, once we get movable locks with nice move-policies, then that will be super-powerfull!

                          Cheers,



                          sacha

                          • 10. Re: Concurrency in SFSB Replication
                            belaban


                            > When you say SFSB state is replicated by a
                            > synchronous broadcast do you mean:
                            > 1) The sender bradcasts the state to the primary
                            > partition, waits to get an acknowledgement from all
                            > the nodes and then returns the reply to the client.


                            Yes, #1. But replace 'state' with 'message' and note that we do not implement a primary partition approach.

                            Bela