1 2 Previous Next 22 Replies Latest reply on Jan 14, 2008 10:24 AM by brian.stansberry Go to original post
      • 15. Re: Implicit marshalled values - a better way of handling re
        brian.stansberry

         

        "jason.greene@jboss.com" wrote:

        This is an example of how it could possibly work with entity caching:
        /entities/[TYPE]/[HASHCODE]
         KEY1 -> UUID1
         KEY2 -> UUID2
         KEY3 -> UUID3
        /entities/[TYPE]/[HASHCODE]/[UUID1]
         [FIELD_NAME] -> [FIELD_VALUE]
         ...
        /entities/[TYPE]/[HASHCODE]/[UUID2]
         [FIELD_NAME] -> [FIELD_VALUE]
         ...
        /entities/[TYPE]/[HASHCODE]/[UUID3]
         [FIELD_NAME] -> [FIELD_VALUE]
         ...
        


        So a given entity requires 2 node lookups, no matter the number of collisions. There will, of course, be contention on the hashcode, but this is expected.


        With Hibernate entity caching there's no second node, as the value is a single Object[] representing the entity fields.

        Some problems with this:

        1) Can lead to deadlocks: node A updates entity 1, node B updates entity 2; they resolve to the same hashcode. Even with your UUID approach you can have that problem if A and B are inserting.

        2) If you don't have two nodes per entity, putForExternalRead no longer works properly, since it uses the presence of the node to decide to quickly abort. That may be trivial to solve; I don't know.

        3) With the 2 node solution, where does the UUID come from? These aren't sessions; each node in the cluster would need to come up with the same UUID for a given key.

        Sorry, still jet-lagged; I can only raise problems today; no solutions :(

        • 16. Re: Implicit marshalled values - a better way of handling re
          jason.greene

           

          "bstansberry@jboss.com" wrote:

          With Hibernate entity caching there's no second node, as the value is a single Object[] representing the entity fields.


          Even better, so in that case, you can just set the value to be Object[].


          Some problems with this:

          1) Can lead to deadlocks: node A updates entity 1, node B updates entity 2; they resolve to the same hashcode. Even with your UUID approach you can have that problem if A and B are inserting.


          Thats not a deadlock, the updates would be serialized though. For a deadlock to occur, you have to update multiple entities in the same cache transaction in a different order, and even in that condition it would just be a timeout. The locking granularity is only as good as the hashcode algorithm, but I think that is reasonable. We would also need to detect the lack of a hashcode function, and provide our own (since the default is based off of identity, which does not work in a cluster).


          2) If you don't have two nodes per entity, putForExternalRead no longer works properly, since it uses the presence of the node to decide to quickly abort. That may be trivial to solve; I don't know.


          I believe this can be solved by modifying putForExternalRead to not only check for node existence, but also key existence.


          3) With the 2 node solution, where does the UUID come from? These aren't sessions; each node in the cluster would need to come up with the same UUID for a given key.


          In the 2 node solution, which does not seem necessary, the UUID does not need to be known by all nodes. It is generated once, for the entire lifespan of the second node. After that point all gets use the entity key, which is known, to retrieve the UUID that is stored as the value of the key on the first node. That UUID is then used to construct the FQN to the node containing the data.

          Sorry, still jet-lagged; I can only raise problems today; no solutions :(


          Keep them coming!

          • 17. Re: Implicit marshalled values - a better way of handling re
            brian.stansberry

             

            "jason.greene@jboss.com" wrote:
            Thats not a deadlock, the updates would be serialized though.


            Sure you can deadlock. When I said "node A" and "node B" I meant different cache instances in the cluster. The both acquire local write locks on the same node in the tree, insert/update their key, then try to acquire the WL globally as part of tx commit. Fails.

            Assume pessimistic locking here (which may not be an issue if we do this far enough in the future, but partly I'm thinking about whether I want to try it this way now.)

            I believe this can be solved by modifying putForExternalRead to not only check for node existence, but also key existence.


            Yeah, that's the solution; just not sure how simple that is, since once you start looking inside nodes you might have to start thinking about locking issues etc that don't exist if you just check node existence. But, it might be trivial.

            In the 2 node solution, which does not seem necessary, the UUID does not need to be known by all nodes. It is generated once, for the entire lifespan of the second node.


            Let's pretend a bit that the 2 node solution is necessary, in case it leads somewhere. :) You can have concurrent putForExternalRead calls on different cache instances, each of which would store a different UUID for the same entity. You'd end up with two copies of the entity in the cache.

            Hmm -- actually you'd get a weird effect where the PFER call for inserting the key/uuid would be aborted when propagated (since the key already exists on the remote node) but the PFER for the uuid node would succeed.

            OK, let's ignore the 2 node solution. ;) Lot's of problems like that; weirdness when Hibernate suspends transactions, but now we're dealing with doing multiple cache writes.

            Keep them coming!


            With OL, we'd have versioning problems, since the version is applied to the node, not the key/value pair. 2 node solution rises from the dead....


            Architecturally, probably cleaner to have a cache per SessionFactory, with the default classloader for deserialization being the deployment's classloader. Seems the only negative to that is if a tx spans session factories, which is probably not a common case.

            • 18. Re: Implicit marshalled values - a better way of handling re
              jason.greene

               


              Sure you can deadlock. When I said "node A" and "node B" I meant different cache instances in the cluster. The both acquire local write locks on the same node in the tree, insert/update their key, then try to acquire the WL globally as part of tx commit. Fails.


              Ok right, with sync replication a lock timeout could occur on simultaneous commit including modifications of the same node.


              Assume pessimistic locking here (which may not be an issue if we do this far enough in the future, but partly I'm thinking about whether I want to try it this way now.)


              This is a problem for all forms of locking. Even with O/L, since a WL is aquired in prepare We really should take a look at handling this condition better. I will start a separate topic on that.

              Collisions should not be common since they would require simultaneous update on an identical hash code, so with a reasonable timeout should be ok.


              Let's pretend a bit that the 2 node solution is necessary, in case it leads somewhere. :) You can have concurrent putForExternalRead calls on different cache instances, each of which would store a different UUID for the same entity. You'd end up with two copies of the entity in the cache.


              Yes, that is possible, since the operations are async and not in the same TX. Hopefully eviction would catch that scenario. You could further reduce the occurence by periodically checking the number of subnodes with the number of key entries. If they are different, purge the dups.


              Hmm -- actually you'd get a weird effect where the PFER call for inserting the key/uuid would be aborted when propagated (since the key already exists on the remote node) but the PFER for the uuid node would succeed.


              Now that is an interesting scenario. Oh how I love PFER and the problems it causes ;) The above (periodic cleanup) solution should work here too.


              OK, let's ignore the 2 node solution. ;) Lot's of problems like that; weirdness when Hibernate suspends transactions, but now we're dealing with doing multiple cache writes.


              Right, it sounds like 1 node is better anyway. The write problems still exist today. The only difference is that it could occur more frequently if there are a large number of writes to non-primitive key objects that have the same hash code.

              Keep them coming!

              With OL, we'd have versioning problems, since the version is applied to the node, not the key/value pair. 2 node solution rises from the dead....


              Ugh. Yes. In general I don't think the cache node version should be defined by the app to begin with. Is there any reason why the "version" can't be an application property? Let's say that Object[] becomes a class that contains String version and Object[]


              Architecturally, probably cleaner to have a cache per SessionFactory, with the default classloader for deserialization being the deployment's classloader. Seems the only negative to that is if a tx spans session factories, which is probably not a common case.


              But then we are back to region based marshaling, and allowing custom types in the fqn, which is very broken.


              • 19. Re: Implicit marshalled values - a better way of handling re
                manik

                Whew - talk about a healthy response to a post! :-)

                First though, back on to the main topic of marshalled values - Brian, I hope to have this in 2.1.0.CR3, which means you will have it for JBoss AS 5 clustering. I hope to have it enabled by default, too - let me know if this causes any issues with you.

                So back to the "new" topic of custom objects in Fqns! ;)

                I think a PFER check on a node attribute can be done just as easily as with a node - the current PFER code does a peek() and tests for a null return value. This could then do a getDirect() for specific attribs.

                I'm not entirely convinced about the UUID approach for entities though. Entities already have an identity - a primary key. Could we not use the entity's PK instead of UUIDs? Surely every entity PK must be mappable to DB primitives in some way, and if it is a composite PK, could we not combine the components in a Collection? This would ensure that the same ID is used cluster-wide and prevent collisions/lock timeouts as well as the need for a "dupe cleanup" thread...

                • 20. Re: Implicit marshalled values - a better way of handling re
                  galder.zamarreno

                  A PK approach seems to fit the purpose better than the UUID one as it's a natural thing for entities. UUID across a cluster might be achievable using java.util.UUID but it's likely to be slower than the PK approach.

                  FYI: Here's a comparison of UUID generators:
                  http://johannburkard.de/blog/programming/java/Java-UUID-generators-compared.html

                  • 21. Re: Implicit marshalled values - a better way of handling re
                    manik

                    Hmm, not including the MarshalledValues patch in CR3. It needs more thought. My current implementation calculates (and caches) hashcodes off the deserialized instance, so when this is replicated, if the key in an attrib map is a MarshalledValue, then hashcode() is called when putting this key in a node and this forces deserialization.

                    Beats the purpose of lazy deserialization as well as re-introduces the need for region-based marshalling since this deserialization is done by the JGroups thread that deals with the incoming replication rather than a user thread on the remote cache.

                    Note that this is only with using MarshalledValues to wrap keys. Works fine for values.

                    • 22. Re: Implicit marshalled values - a better way of handling re
                      brian.stansberry

                      No problem not being in CR3. For now, the biggest benefit to me from this (saving memory on the values) I can implement myself at the AS level. If we rush it into JBC before we've thought it all through and there's a problem, then I'm kind of hosed for AS 5 beta4 and Hibernate 3.3. So caution is good. :)

                      1 2 Previous Next