11 Replies Latest reply on Mar 8, 2010 2:03 AM by belaban

    Scalability of CirrAS/JBossAS Cluster


      Not sure if this is the right forum.


      So with the CirrAS images you provided and the computing power of Amazon EC2 it's easy to build a JBoss cluster with 1-n nodes. But when working with stateful applications a cluster always has to manage things like session replication between the nodes.


      What I would like to know is what number of nodes makes sense - meaning: when does the effort of managing session replication exceed the benefit of new nodes in my cluster? Probably there is no universally valid answer to this as the number of nodes depends on the clustered application - so what is the best way to determine the maximum number of nodes which make sense for my application?




        • 1. Re: Scalability of CirrAS/JBossAS Cluster

          There is no direct answer, but I did watch a good talk by Bela Ban on this. The context was in the mod_cluster "domains" - having domains of cluster nodes that are kind of independent from other domains (so within a domain you can have the replication you want/need, but not across domains). At the end of it, he sees 100 nodes as generally being a limit where you would want to consider splitting it into domains (ie overhead of replication gets annoying).


          Unfortunately the presentation isn't public (at the moment) but hopefully that will change soon, as it was a brilliant talk.

          • 2. Re: Scalability of CirrAS/JBossAS Cluster
            Thanks for your answer Michael. I'm gonna ask Bela Ban if can provide me the presentation.
            • 3. Re: Scalability of CirrAS/JBossAS Cluster

              Hi Matthias,


              there are 2 types of redundancy: one's called replication and the other one distribution. Replication replicates every data to every other node in the cluster, so the cluster size is bounded by (1) the number of nodes and (2) the average data size of each node. E.g. 10 nodes with an average data size of 100MB means that every node has to store data of 1GB on average.


              Distribution (as done for example in Infinispan) means that a data element is stored N times in the entire cluster. So if every data element is stored 2 times, then the average data stored at any given node is 200MB (versus the 1GB for replication).


              However, if 2 nodes storing the 2 data element die at the same time, your data is gone. So, distribution is a tradeoff between data usage and data redundancy.


              • 4. Re: Scalability of CirrAS/JBossAS Cluster
                yes sorry I should have mentioned that I assumed DIST mode if domains are used - and also it seems to be becoming the "preferred default" in Infinispan. I imagine in the future distribution mode with N=3 would be pretty decent redundancy inside a domain?
                • 5. Re: Scalability of CirrAS/JBossAS Cluster

                  The cool thing in Infinispan is that different caches can be created with different Ns, and you place your data in a cache according to your reliability requirements.


                  Re: replication versus distribution: I guess if you're cluster is smallish and/or your data small, then I'd prefer replication because reads are way faster (because they're always local) than in DIST, where we might have to do a sync round trip if the data is not in the L1 cache.


                  So, for example, if you have 10 nodes and every node has 10MB worth of data, then I'd go for replication. If you have 50 nodes, and every node generates 200MB worth of data, then I'd pick distribution.

                  • 6. Re: Scalability of CirrAS/JBossAS Cluster
                    good points - would having sticky session routing in the mod_proxy help with avoiding remote cache reads ? (not sure if it is fine grained enough to know which of the "N" nodes to direct a sessions request to) - so each of the N nodes that has a copy of the data has it in the L1 right? or am I getting confused?
                    • 7. Re: Scalability of CirrAS/JBossAS Cluster

                      No, the load balancer has no idea about how Infinispan distributes its data. E.g. if we have {A, B, C} and N=1 and data element D is stored on B.


                      Say the load balancer routes a session S to A, then A will ask B for the data the first time it doesn't find it in its L1 cache. Subsequently, A will always return D immediately, unless and until D is invalidated (e.g. because it was updated).

                      • 8. Re: Scalability of CirrAS/JBossAS Cluster

                        At some point, we should brainstorm a bit about whether it makes sense to add pluggable routing (in C) to the load balancer. If we had the same consistent hashing algorithm on the load balancer (mod-cluster) as in Inifinispan, the load balancer could reduce the number of additional round trips to fetch data not in the L1 caches...


                        Applied to HTTP sessions, individual attributes of a session are stored on different nodes, so 'stickiness' here would be broken down to attributes, not an entire session. As an alternative, a consistent hashing algorithm could used the jsessionid as input, so all attributes of a given session would be colocated.


                        This needs more discussion (probably on the mod-cluster forum), I'll ping Brian ...

                        • 9. Re: Scalability of CirrAS/JBossAS Cluster
                          Good point, I incorrectly assumed that all HttpSession data would be in one place, not have each attribute (potentially) distributed - is this deliberate, but just by design (ie it is a distributed cache, so naturally data on each attribute stands alone) ? As I would like to know if it was thought of as a good idea, or a side effect (I can't think of why it would be a good idea - but I guess in practice people often put a "value object" in a session, so all state is in one java objects - mostly for type safety when building the app).
                          • 10. Re: Scalability of CirrAS/JBossAS Cluster
                            We already had a discussion on it, last year in Brno, and the wiki is https://community.jboss.org/wiki/Consistent-hashingbasedJBCdatapartitioningwmodcluster
                            • 11. Re: Scalability of CirrAS/JBossAS Cluster

                              That's something to discuss. One could argue, as attrs of a sessions are accessed together, that they should be stored together. OTOH, if the attrs are big, then it might make more sense to distribute them. Distribution of individual attrs adds more complexity to mod-cluster though.


                              We'll discuss this during our Brno meeting next week.