1 2 Previous Next 15 Replies Latest reply on Jun 1, 2006 6:15 AM by manik

    DataGravitation of entire subtrees

    brian.stansberry

      Having some trouble integrating BR into HttpSession repl due to some quirks in data gravitation.

      For ATTRIBUTE granularity, a session is stored in 2 nodes -- parent node "metadata", whose name is the session id, and which stores session metadata, and child named ATTRIBUTE, which stores the attributes in its data map.

      2 server cluster, servers A and B; session owned by A.

      Upon failover, get() is called against the Fqn of "metadata". This causes "metadata" to be gravitated. What happens next depends on how I configure "dataGravitationRemoveOnFind".

      If true, a remove call is sent around the cluster. A receives it and removes the metadata node as well as the ATTRIBUTE node. When B then wants to access the ATTRIBUTE node, it needs to gravitate it, but can't because it's been removed!

      If false, an evict call is sent around the cluster. A evicts the metadata node's data, but leaves the node, because it still has child ATTRIBUTE. B then tries to access ATTRIBUTE, can't find it, and the data gravitates from A. Another evict is sent over the cluster; this time A removes the ATTRIBUTE node, as it has no children.

      Problem here is the empty metadata node is left on A. This screws things up if that Fqn is ever accessed again on A. The presence of the empty node breaks data gravitation!

      I was able to hack around this for ATTRIBUTE granularity, but in a way I'm really not happy about. Also I fear that this will be a big problem with FIELD granularity, where the node structure is much deeper. That I still have to test.

      In general, leaving empty nodes littered around is not good. For BR to work properly, it's critical that nodes in the main tree only exist in the cache that owns them.

      One concept I thought of is adding a switch whereby entire subtrees could be gravitated in one go. Perhaps returned as an array of NodeData objects, similar to state transfer. If that could be done, then having "dataGravitationRemoveOnFind" set to true would work fine -- the remove call would be no problem as the child nodes would all be moved.

      With such a switch though, an Option to disable DataGravitation for individual calls would certainly be needed so the application can safely do things like put("/JSESSION/localhost/webapp1") to create the root node for an app's sessions without worrying about the whole tree gravitating. But, I think that Option is needed anyway.

        • 1. Re: DataGravitation of entire subtrees
          manik

           


          One concept I thought of is adding a switch whereby entire subtrees could be gravitated in one go. Perhaps returned as an array of NodeData objects, similar to state transfer. If that could be done, then having "dataGravitationRemoveOnFind" set to true would work fine -- the remove call would be no problem as the child nodes would all be moved.


          I agree this needs to be there. Not just as a switch, but I would say as default behaviour. Why would anyone not want the entire subtree gravitated?

          • 2. Re: DataGravitation of entire subtrees
            manik

             


            With such a switch though, an Option to disable DataGravitation for individual calls would certainly be needed so the application can safely do things like put("/JSESSION/localhost/webapp1") to create the root node for an app's sessions without worrying about the whole tree gravitating. But, I think that Option is needed anyway.


            I can't see why such an option is needed. If you are doing a put like that, you should gravitate that node - if it exists elsewhere in the cluster - to the instance you are doing the put on. In the end the whole tree will not be gravitated, just /JSESSION/localhost/webapp1 and its children.

            • 3. Re: DataGravitation of entire subtrees
              manik
              • 4. Re: DataGravitation of entire subtrees
                manik

                Just been looking at other potential use cases for such an Option (as you brought up in http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3946773#3946773)

                Created a task for this: http://jira.jboss.com/jira/browse/JBCACHE-637

                • 5. Re: DataGravitation of entire subtrees
                  brian.stansberry

                   

                  "manik.surtani@jboss.com" wrote:

                  With such a switch though, an Option to disable DataGravitation for individual calls would certainly be needed so the application can safely do things like put("/JSESSION/localhost/webapp1") to create the root node for an app's sessions without worrying about the whole tree gravitating. But, I think that Option is needed anyway.


                  I can't see why such an option is needed. If you are doing a put like that, you should gravitate that node - if it exists elsewhere in the cluster - to the instance you are doing the put on. In the end the whole tree will not be gravitated, just /JSESSION/localhost/webapp1 and its children.


                  Some background on this particular use case:

                  In the case of /JSESSION/localhost/webapp1, the JBossCacheManager creates that node when the webapp is deployed. It's creating it as the parent node under which individual session subtrees will be placed. It doesn't want to gravitate data, it just wants to create a node under which it can place session subtrees when it starts taking requests. If it gravitates data, each server in the cluster will send back the sessions it's handling -- the data will be different from each server. Only the first response will be integrated -- just some of the sessions. Then when the cleanup call goes out all the other servers will discard their sessions. Oops!

                  And actually, creating /JSESSION/localhost/webapp1 would first cause gravitation of /JSESSION, so now ALL sessions would be lost except for those from the first respondent. Double oops!

                  • 6. Re: DataGravitation of entire subtrees
                    brian.stansberry

                     

                    "manik.surtani@jboss.com" wrote:

                    One concept I thought of is adding a switch whereby entire subtrees could be gravitated in one go. Perhaps returned as an array of NodeData objects, similar to state transfer. If that could be done, then having "dataGravitationRemoveOnFind" set to true would work fine -- the remove call would be no problem as the child nodes would all be moved.


                    I agree this needs to be there. Not just as a switch, but I would say as default behaviour. Why would anyone not want the entire subtree gravitated?


                    I can't think of a reason that gravitating the entire subtree wouldn't be wanted -- but my brain is lacking in creative juice today ;)

                    If we come up with a reason and leave in place the ability to gravitate just one node at a time, we have a problem with getChildrenNames(). Currently we gravitate the data map, but the names of a node's children is also a piece of critical data -- without it an app can't walk the tree. FIELD granularity session repl currently breaks because of this. Gravitating the entire subtree will fix the session repl case.

                    Let's hope we can't think of a good case for not gravitating the whole subtree and then the getChildrenNames() issue goes away. :-)

                    • 7. Re: DataGravitation of entire subtrees

                      Will graviation based on "region" works better? It is a sub-tree but more well-defined. Things is of course now gravitation will need to have knowledge of "region".

                      The downside that I can see for gravitating the whole sub-tree is if I accidentally get a root node, that can mean lots of data being migrated during the first get (whenever it is).

                      • 8. Re: DataGravitation of entire subtrees
                        brian.stansberry

                        Can you give an example of what you mean?

                        In the http session replication case, the "region" is too big -- regions for http session repl are all of the nodes associated with a particular webapp. We just want to gravitate a particular session, which is smaller.

                        You're absolutely right though -- apps that are working with a cache that uses data gravitation need to be very careful to not accidentally gravitate data. Proper use of the new "suppressDataGravitation" Option is very important.

                        • 9. Re: DataGravitation of entire subtrees
                          manik

                          Looking at fixing this today.

                          Rather than a set of NodeData objects, how about if the remote call returns a byte[] generated by calling cache._getState()?

                          Seems to be a clean and easy way to gravitate a subtree, but I do have one concern: on the receiving end, how can I call setState() such that I am able to attach the new state to the root f the tree (take ownership) and then ensure all interceptors fire for Fqns involved in this state, as thoug I had done a put()?

                          • 10. Re: DataGravitation of entire subtrees
                            manik

                            Already implemented as a list of NodeData objects, BTW. I was just wondering whether raw byte[] state would be more efficient.

                            • 11. Re: DataGravitation of entire subtrees
                              brian.stansberry

                               

                              "manik.surtani@jboss.com" wrote:
                              Seems to be a clean and easy way to gravitate a subtree, but I do have one concern: on the receiving end, how can I call setState() such that I am able to attach the new state to the root f the tree (take ownership) and then ensure all interceptors fire for Fqns involved in this state, as thoug I had done a put()?


                              You really can't, at least not in a generic sense. _setState completely bypasses the interceptor chain.

                              If you're interested in ensuring certain things done by the interceptor chain are done, we can perhaps work that into what _setState does. The one that comes to mind is TreeCacheListener notifications.

                              • 12. Re: DataGravitation of entire subtrees
                                brian.stansberry

                                I think we are going to have to transfer a byte[] rather than the NodeData directly. The NodeData may contain objects that JGroups cannot deserialize (no proper classloader). So we need to pass back the byte[] and deserialize it in the DataGravitatorInterceptor, after we make sure we have set the TCCL correctly.

                                Boy, it sure sounds like a state transfer, doesn't it?

                                • 13. Re: DataGravitation of entire subtrees
                                  manik

                                  Hmm, my problem with that is we don't end up firing the interceptors properly. Which means updating cache loaders and replication to buddy backups don't happen.

                                  And even then, doesn't serialization/deserialization happen in the EnhancedTreeCacheMarshaller? JGroups just deals with a byte[] generated/handled by the ETCM on either end.

                                  • 14. Re: DataGravitation of entire subtrees
                                    brian.stansberry

                                     

                                    "manik.surtani@jboss.com" wrote:
                                    Hmm, my problem with that is we don't end up firing the interceptors properly. Which means updating cache loaders and replication to buddy backups don't happen.


                                    (Palm smacking head ) Doh!! Of course; just integrating the data would break as much as it would fix.

                                    The idea of changing StateTransferIntegrator to do put() calls instead of directly building the subtree is sounding better and better though. But not for 1.4.0 :)

                                    "manik.surtani@jboss.com" wrote:
                                    And even then, doesn't serialization/deserialization happen in the EnhancedTreeCacheMarshaller? JGroups just deals with a byte[] generated/handled by the ETCM on either end.


                                    Not sure of the context, so two replies:

                                    1) For state transfer, ETCM doesn't really do anything, as the main payload for the MethodCall is a byte[]. StateTransferIntegrator then works with the marshalling RegionManager to get the proper classloader to deserialize the byte[].

                                    2) For RPC calls that pass data around, ETCM comes into play on the node that receives the RPC *request*. RpcDispatcher uses the ETCM to deserialize the MethodCall. Then that node generates a response. If the response contains data that needs special deserialization on the other end, RpcDispatcher provides no hooks to register something like ETCM to handle that. So, the only safe way to pass such data back is as a byte[]. The application (i.e. JBC) then has to deal with the byte[].

                                    Most calls in JBC are pushing data with the MethodCall and the response is just null. There are only 2 cases where the data is coming back with the response -- partial state transfer and data gravitation. For partial state transfer, the data already comes back as a byte[].

                                    The real solution here is to add hooks to JGroups to allow marshalling of RPC responses.

                                    1 2 Previous Next