9 Replies Latest reply on Feb 15, 2010 3:42 AM by goldmann

    Multi-tenanting and CirrAS/StormFolio - design thoughts

    michaelneale

      So if using CirrAS (and StormFolio) we stand up a cluster of app servers + a front end appliance - how can we best support multi tentanting?

       

      by this I mean that in a PaaS cloud, we may (often) want to have several peer apps running in the one cluster  (of course we may not, some apps may need to run stand alone - but that is decided by something like cooling tower).

       

      Mostly this means allowing the database back end to have its own schema/table space for each app.

       

      On the application server tier, we can either have each deployed side by side in the one app server or

      startup a separate instance of an app server for each app (different ports) so its like a parallel cluster.

       

      Where things get gnarly is on the front end appliance.

      As we have apache with mod_cluster, it is doable:

       

      app1.service.com, app2.service.com and so on can be routed accordingly (potentially to different ports on the app servers). DNS can then point multiple apps to the one front end appliance, and its routing rules kick in from there (of course if we want to move an app off, we can, update its front end appliance address).

       

      Thoughts?

        • 1. Re: Multi-tenanting and CirrAS/StormFolio - design thoughts
          goldmann
          Random ramblings below.

          So if using CirrAS (and StormFolio) we stand up a cluster of app servers + a front end appliance - how can we best support multi tentanting?

           

          by this I mean that in a PaaS cloud, we may (often) want to have several peer apps running in the one cluster  (of course we may not, some apps may need to run stand alone - but that is decided by something like cooling tower).

          A really good question. Multi tenancy is a quite big issue we're facing here. Bob presented in Boston last time a few PaaS types:

          • In type 1 (current CirrAS status) we have two options:
            • trusted environment: we can host multiple apps withing one cluster.
            • untrusted environment: we can host multiple apps within different clusters.
          • In type 2 (CirrAS + CT target?) we should be able to do this more easily. Type 2 is a Type 1 with isolated and managed set of services. In Type 2 we should be able to add or remove services and specify restrictions, isolation (CT work?).

           

          I'm thinking now that probably an app should be packaged into a software appliance that could be deployed on PaaS platform. This package (app + metadata) should contain information about the deployment such as: tenancy information (we're in cluster XYZ and don't want to share resources with other apps) and datasource descriptors too (assuming we're deployng Java EE  stack).

          Mostly this means allowing the database back end to have its own schema/table space for each app.

           

          We could specify also datasource descriptors in out metadata: ie. encapsulate descriptors from app itself and change the database name, table prefixes, etc. Of course we're talking now about realtional databases. If we use Type 2 PaaS we schould have also a non-query datastore available as a REST service. This would be easier to arrange IMO. How about exposing infinispan grid and then make JPA query interface available? Of course in that case we still must be able to backup data in some database (power outage and we're in black hole...).

           

          But, back to the earth.

           

          What can we do now? I think the easiest way to have multi tenancy with isolation now is to run Configuring Multiple JBoss Instances On One Machine on one node (using ports). If we're not interested in isolation (we're trustin other apps) we can deploy it side by side.

          Where things get gnarly is on the front end appliance.

          As we have apache with mod_cluster, it is doable:

           

          app1.service.com, app2.service.com and so on can be routed accordingly (potentially to different ports on the app servers). DNS can then point multiple apps to the one front end appliance, and its routing rules kick in from there (of course if we want to move an app off, we can, update its front end appliance address).

          How about specyfying virtual hosts on back-end? It should be injected to front-end and all requests to that domain should be routed to specific back-ends. We could add virtual host definition to metadata packaged with our app and inject a virtual host definition using JMX calls to JBoss AS.

          • 2. Re: Multi-tenanting and CirrAS/StormFolio - design thoughts
            michaelneale

            I'm thinking now that probably an app should be packaged into a software appliance that could be deployed on PaaS platform. This package (app + metadata) should contain information about the deployment such as: tenancy information (we're in cluster XYZ and don't want to share resources with other apps) and datasource descriptors too (assuming we're deployng Java EE  stack).

            Actually - the app should not be in the appliance itself - I see the appliance as the shell, empty, ready to take applications and meta data. Starting up an appliance/building a cluster - that is just like starting up an app server on your laptop - things are ready, but it does nothing until you deploy something to it. And of course, you can undeploy, and remove the app.

             

            As we are talking type 2 here (somewhat) - CT would have worked out if it can multi tenant or not - the app cluster just has to be able to take a new app + settings.

             

             

            We could specify also datasource descriptors in out metadata: ie. encapsulate descriptors from app itself and change the database name, table prefixes, etc. Of course we're talking now about realtional databases. If we use Type 2 PaaS we schould have also a non-query datastore available as a REST service. This would be easier to arrange IMO. How about exposing infinispan grid and then make JPA query interface available? Of course in that case we still must be able to backup data in some database (power outage and we're in black hole...).

            Yes - well we can assume there would be some data in the application bundle which would tell us how to create/populate a database, migrate if needed etc... we would use 1 table space per app I guess (named after the app). I suppose it is possible to have 1 database process per app, but I don't think that would work that well. As for Infinispan/REST - yes that will happen, but I think practically speaking for a reasonable future RDBMS services will be expected.

             

             

            How about specyfying virtual hosts on back-end? It should be injected to front-end and all requests to that domain should be routed to specific back-ends. We could add virtual host definition to metadata packaged with our app and inject a virtual host definition using JMX calls to JBoss AS.

             

            I guess that could work - so the front end doesn't have to know too much about it then?

             

            Say we have 3 apps in a cloud: hr.company.com, finance.company.com, wiki.company.com (1 of which could be bespoke, the others off the shelf) - it should be possible to have all these running in the same cluster/instance if need be - as 3 separate wars. Of course, the time may come where the hr app outgrows, and moves to a separate cluster - in which case the name remains the same (and either the front end appliance now directs traffic, or the DNS is changed to point to a new front end appliance).

             

            I will attach some pictures to try and explain some options.

            • 3. Re: Multi-tenanting and CirrAS/StormFolio - design thoughts
              michaelneale

              Some drawings..

               

              First one: an app cluster contains front end appliance and app servers (database is outside): ClusterArch.png

               

              Second, front end is outside of cluster, possibly shared across diff clusters...

              in all cases, database appliances are separate. And appliances are not specific to a user application (war/ear etc).

               

              ClusterArchAlt.png

              thoughts?

              • 4. Re: Multi-tenanting and CirrAS/StormFolio - design thoughts
                goldmann

                I'm thinking now that probably an app should be packaged into a software appliance that could be deployed on PaaS platform. This package (app + metadata) should contain information about the deployment such as: tenancy information (we're in cluster XYZ and don't want to share resources with other apps) and datasource descriptors too (assuming we're deployng Java EE  stack).

                Actually - the app should not be in the appliance itself - I see the appliance as the shell, empty, ready to take applications and meta data. Starting up an appliance/building a cluster - that is just like starting up an app server on your laptop - things are ready, but it does nothing until you deploy something to it. And of course, you can undeploy, and remove the app.

                 

                As we are talking type 2 here (somewhat) - CT would have worked out if it can multi tenant or not - the app cluster just has to be able to take a new app + settings.

                I agree. I think you missread my previous post. I was thinking about two appliance types:

                1. a VM – image you can deploy with all the software installed (httd, JBoss, etc)
                2. a software appliance: your app + metadata that would describe this application and maybe some additional data with the PaaS setup information (cluster name, isolation requirements, etc).

                 

                I was talking about 2nd one. More, CT could create the metadata description and client would only pass the EAR or WAR file to CT to deploy it. I think sooner or later there will be a need to atach some more info about deployment.

                How about specyfying virtual hosts on back-end? It should be injected to front-end and all requests to that domain should be routed to specific back-ends. We could add virtual host definition to metadata packaged with our app and inject a virtual host definition using JMX calls to JBoss AS.

                 

                I guess that could work - so the front end doesn't have to know too much about it then?


                I think that's the goal: mod_cluster shouldn't know what's on the  back-end. It's easier to manage.

                • 5. Re: Multi-tenanting and CirrAS/StormFolio - design thoughts
                  goldmann
                  I would say +1 for load balancer(s) outside the cluster. Load balancer should be considered always as an external box outside the cluster IMHO.
                  • 6. Re: Multi-tenanting and CirrAS/StormFolio - design thoughts
                    michaelneale

                    I agree. I think you missread my previous post. I was thinking about two appliance types:

                    1. a VM – image you can deploy with all the software installed (httd, JBoss, etc)
                    2. a software appliance: your app + metadata that would describe this application and maybe some additional data with the PaaS setup information (cluster name, isolation requirements, etc).

                     

                    I was talking about 2nd one. More, CT could create the metadata description and client would only pass the EAR or WAR file to CT to deploy it. I think sooner or later there will be a need to atach some more info about deployment.

                     

                    Right- when I see "appliance" I think of something that has to come out of boxgrinder - whereas the end users ears/wars + conf data would NOT be put through box grinder - but added after the fact.

                     

                    Yes - the user would need to provide some config data for CT to use to setup the app servers via APIs (or it will have to infer them from the

                     

                    I would say +1 for load balancer(s) outside the cluster. Load balancer should be considered always as an external box outside the cluster IMHO.

                    OK great - yes - means they can be shared for lots of clusters, potentially (I have no idea how practical it is interms of load).

                    • 7. Re: Multi-tenanting and CirrAS/StormFolio - design thoughts
                      goldmann

                      Yes - the user would need to provide some config data for CT to use to setup the app servers via APIs (or it will have to infer them from the

                      Question is it should be done by the user or CT should "pack" user app into a application appliance?

                      I would say +1 for load balancer(s) outside the cluster. Load balancer should be considered always as an external box outside the cluster IMHO.

                      OK great - yes - means they can be shared for lots of clusters, potentially (I have no idea how practical it is interms of load).

                      Of course we could (or maybe should) have multpile front-end's. Current CirrAS deployment has support for multiple front-end's out of the box.

                       

                      I'm currently fighting with addign additional front-end's without the need to restart JBoss AS on the back-end. Probably this cannot be done easily, but finger crossed.

                      • 8. Re: Multi-tenanting and CirrAS/StormFolio - design thoughts
                        michaelneale

                        Question is it should be done by the user or CT should "pack" user app into a application appliance?

                        By this I mean it is really a descriptor, so user provides a (new) app - inside that app is a "cloud descriptor" (or perhaps it can be inferred) describing the apps name, what data sources it needs, etc... CT then will create a cluster/DB based on that if needed, and then deploy the application to it (or perhaps there is already a cluster/DB suitable that can be multi tenanted, in which case CT will target it).

                        • 9. Re: Multi-tenanting and CirrAS/StormFolio - design thoughts
                          goldmann

                          By this I mean it is really a descriptor, so user provides a (new) app - inside that app is a "cloud descriptor" (or perhaps it can be inferred) describing the apps name, what data sources it needs, etc... CT then will create a cluster/DB based on that if needed, and then deploy the application to it (or perhaps there is already a cluster/DB suitable that can be multi tenanted, in which case CT will target it).

                          Maybe we're going in the wrong direction. How about this. Every user of our our platform will have DB credentials given afteraccount set up. We'll allow access to our db cluster service for this particular user only for those credentials. Using that scenario our cloud descriptor could only have cluster name. We can mix this with client ID to not interfere with other clients. In that case we'll run new JBoss instances with this cluster name as a domain name. See Bela's presentation.

                           

                          We would have multi tenancy (we could run multiple JBoss AS on one node using ports) and isolation (apps from different users will be placed in a different cluster – data collision with other users will not happen because we're completely isolated).

                           

                          WDYT?