13 Replies Latest reply on Nov 10, 2010 2:12 PM by Todd Morrison

    Seam scalability estimation

    Milan Szabo Newbie

      Hello, I have done many estimation about scalability of my project, but still I think that some guess or advice from more experienced developers would be helpful especially because in my project I cannot use some multi-client simulators.


      I will try to explain what is it all about..


      Project - It is a free ajax-based web game where client manipulate mainly with buttons not forms. So not a classic form application but something like click - action - ajax response. Client works at only ONE page most of the time where main panel is rerendered according to navigation button clicked. I think that it is better to rerender only about 60% of page (40% is content which is always shown) than a whole page.


      I have spent a lot of time and my effort to optimization and I think I have really optimized system .. but maybe still not enough. I will explain architecture now.


      In my game are all players always in game ant their objects are singletons in a class which holds their static list. Only players currently logged in have their own session, but after logoff their avatar remains alive on the server side.


      There are several asynchronous methods which performs user actios (actions here are in real time .. not click and done) In my whole application IS NOT ANY DB QUERY expcept one for world save (every 15 minutes). So DB queries count DOES NOT increase with number of clients. Simply I am trying to hold objects in memory as a singletons because I dont need to work with lists of hundred thousands of objects.


      Codebooks - codebooks are initialized at server startup and then available as singletons AND as a parent of some instantiated item. All items in my game (which I think will be the most numerous object) has that parrent with static data and it is singleton for all items of same type and only a few instance-related data has this item instance on its own.


      Clients just use some A4J buttons to manipulate with their avatar (doing actions, using inventory, travelling and so on so again not classic forms). Client is refreshed with A4J poll (only a little part of page) and if the situation allows me to controll object changes, then i use A4J PUSH (which is lot more effective) In total I have like 1 POLL/sec (up to 2KB response) and about 1 PUSH/sec (maybe up to 5KB, but only when something really changes else 0KB). If I have tested it on my own I have generated response code from server in amount of 2KB/sec per client.


      On server-side I dont use long-running conversations because I dont have a need to it - No many-step wizards are there and I think that conversation means some overhead. Not many statefull beans per session, only about 15 with actions and 30 which are only form beans to hold selected info. All are POJOS with Session scope.


      I have done some optimizations on view-side where I dont use richfaces anymore because they render too much code .. I rather wrote my own javascripts for tooltips. Modals are still RF (last thing) but probably I will also use some other approach. I have a server filter which provides controll of caching images,js,css so no problem here.


      What Am I afraid of - there are three things..


      1 - Server load for processing many ajax-requests when more players log in even if no db-queries and only the mandatory amount of code rendered back.


      2 - Server load for processing asynchronous things which simulates world live (no DB, no complicated operations but a LOT of simple ones) these are not connected to JSF at all.


      3 - Server memory usage with 100.000 items game, 2000 avatars in world and 400 players logged in. How much memory could it need to have, I have many singletons used so there should not be terrible increment in memory usage but still I need at lead these numbers of players to make that game funny.


      As I mentoined above I expect about 2000 - 4000 avatars in world (only on background, no web session) where each avatar could have something like 50 objects (but these objects are not complicated mainly primitive data types). I expect about 100 - 500 clients simultaneously which means about 1000 - 2000 requests per second (it will be the hardest problem I think because of JSF and ajax processing).


      So finally .. what is your opinion ? Will be some quad-core 8GB server able to handle that game ? I need to discuss it because I have spent about 1000 hours of development by now and expect 3000hours more .. and I will be frustrated if I see it not working after all of that work and I cannot afford several servers in cluster. I hope that if optimized and not so complicated web game is done in year 2010 on quad cre GHZ many GB server that it SHOULD WORK but everywhere I read some comparisons there Seam seams to be the slowest framework in the world, thats why I am afraid because work in seam with JSF is so nice.


      Thanx in advance for your opinions I will answer any possible question if someting is not clear. I ma not experienced in using s:cache tag for example and maybe it could save a lot of server load.

        • 1. Re: Seam scalability estimation
          Milan Szabo Newbie

          Maybe my topic is too long and discourages everyone from reading and considering :))

          • 2. Re: Seam scalability estimation
            Milan Szabo Newbie

            I have done some tests with my firends and results are really horrible. I dont know where the problem is.


            Our test was configured like that..


            I set to one poll which refresh some part of page (only outputs and 8 action buttons) response size is 1.6KB per request and interval was 100ms so 10 req/client/sec. This part of page is not changed during tests so the same data all the time.


            I have one client in FF for measuring response time by firebug
            My roommate has 6 IEs so totally 70 requests per seconds from 7 clients with 70KB/s of response size (we are on LAN so no problem here). 70KB/s of server response without any background logic I would guess that Intel 486 / 66mhz should be able to handle it and what happens with my Dual Core Intel with 4GB ram ? Lets see..


            With only my client here is 150ms response time which is horrible on its own ! 150ms for 1.6KB of simple HTML in 21 Century ?WTF :( And the most interesting is, that my processor workload is on 5% and it includes browser. Why do I have such slow responses when the processor has reserves. If I click in some complex webpages I could have response of 100KB in less than 50ms and on localhost simple chunk of data needs 150ms ?


            If we run 70 requests/sec then I got response OVER 1 second !! so there really was not 70 requests but only 7 per second which brings my super dupper jboos to process 9KB/s whow employee of the month is here.


            Most interesting is that if we had 5 browsers on then I got response time about 200ms and 50% processor workload and when 7 browsers then response over 1 sec BUT processor workload dropped to 20% How could this even be possible ? Why processor doesnt work and I have to wait more than 1 sec.. I really cannot understand.


            And to be honest .. I have to consider swing GUI for my game .. even if it is less complicated than Ultima Online from 1995 .. it cannot support more than 10 players on dual core PC.


            My PC is notebook maybe server would behave different way but still that notebook is really fast. So where could be problem :-/

            • 3. Re: Seam scalability estimation
              Paul Keogh Novice
              This article by Dan Allen might help ?

              http://www.jsfcentral.com/articles/speed_up_your_jsf_app_1.html
              • 4. Re: Seam scalability estimation
                Milan Szabo Newbie

                This is not applicable to my type of project. This help is focused on classic data table applications .. it is really different from what I have ..


                I have probably found the problem .. but I dont have a solution.


                I have problem if more than 6 clients are connected to my server ! I have response about 100ms from one client, from two, three, four, five .. still good responses with incremental server load. But after I connect from sixth browser then response goes to 400 ms and server load DECREASES DRAMATICALLY .. If i connect 7th browser  then server load is about 5% and responses 2s or more. Here is something wrong in configuration what could it be ?


                5 browsers works great .. good time and server works fine, after I add 6 or 7 then everything screws up AND regardless of ajax poll interval !! It doesnot matter whether I sent 10 per second or 1 per second, results are the same, 5 browsers OK, more KO.

                • 5. Re: Seam scalability estimation
                  Tim Evers Master

                  This is no way to be testing scalability.


                  Testing by trial and error is almost a waste of time.


                  You need to put a profiler on your server. See WHY things are taking so long. A profiler can tell you where the time is being spent, how much memory is being used etc. There's a possibility that you just haven't given enough memory to your jvm and that a lot of time is being spent in garbage collection.


                  But, for the sake of your own sanity use a proper tool to find out the problem.

                  • 6. Re: Seam scalability estimation
                    Milan Szabo Newbie

                    I know about JProfiler but I just cannot believe that with all that optimization I done 6 clients without any objects without any db query just rerendering 6 outputTexts is too much for server. It only takes about 300MB (win task manager) and JVM has 1GB so there is no need for garbage collecting. But it is not important now. I have decided to switch to java Aplets because i just need much more clients and JBoss and Seam just cannot handle it quick enough.

                    • 7. Re: Seam scalability estimation
                      Leo van den berg Master

                      Hi,


                      I saw that you stated that Dan Allen's article is not applicable for you, because it's about datatables. I think that is not entirely correct. Seam is a great framework and it really speeds up development, but there are some things to keep in mind. If you make complex beans with a lot of functionality, a lot of IN and OUT annotations AND you are completely stateless (which basically is every Ajax request), without proper precautions, a bean is instanciated and all injects and outjects are made when you access a method on your bean!!


                      Some tricks:





                      • When you don't need Seam's intervention (in the case of the simple get/set) use the ByPassInterceptor anotation.




                      • Try to split up your bigger beans to prevent continuous in- and outjecting of values which you don't need in a specific method.




                      • Use factories where possible.




                      • Do NOT use the unwrap annotation directly, without a proper factory (caching) companion.




                      • Change to Ajax push if possible



                      Hopefully this helps a bit.



                      Leo


                      • 8. Re: Seam scalability estimation
                        Milan Szabo Newbie

                        Hello I have all beans that are called through ajax as SESSION scoped so only one instance per session. These beans are architectured with in and out in mind so there is really a few of injections. Not more than 3 - 5. And for my test case I dont use complicated logic. Most of data that being displayed are in memory and are singletons so server just return their getters.


                        I have ajax push in my app almost everywhere I have used polls just to simulate server load for more clients.


                        So I really dont think that I miss something but the pain is the fact that processor does not work ! Its load is 50% for 5 clients (10 req/s/client) and then if I add more clients processor load wend DOWN not UP ande response times raise to seconds. That is weird, why processor runs only at 10% with 7 clients and 50% with 5 clients. There must be some wrong configuration I think .. When I have 7 clients with brutal response times and processor load 10% and close one client then everzthing is ok .. response 100ms and processor at 50%. Maybe this is not seam´s fault.


                        But still I think that for game with real-time interaction of more clients is applet more suitable than web-application. It is hard cos I am used to write web apps but maybe it is better to switch approach now than spend lot more time and then be disappointed when 50 real players log in and kill server.

                        • 9. Re: Seam scalability estimation
                          Leo van den berg Master

                          Hi,


                          you really need some additional tools to figure out what is happening, because just staring at the code and console won't help you a lot.


                          Leo

                          • 10. Re: Seam scalability estimation
                            Craig Bensemann Novice

                            Hi,


                            The fact that your cpu load went down suggests that you have contention on a resource in your code or that the system is waiting for something else such as disk access or network access. The fact that you only have 7 clients causing this suggests there is something seriously wrong with your application but as everyone else has suggested you REALLY need to get some tools and to profile your application. Also look at your test client setup. Its quite possible that multi-threading issues on your test client can slow down your performance test and not the server at all.


                            I have written large java applications - 200 million+ requests per day (this particular app was not written in seam as it was before i moved to seam). So I know java profiling and performance testing can be difficult but I also know java (and I beleive seam) is more than capable.


                            Craig

                            • 11. Re: Seam scalability estimation
                              Tim Evers Master

                              Something else you should consider.


                              If you don't take the time to work out why you are having the performance problems now then switching to an applet may in fact not help you at all.


                              • 12. Re: Seam scalability estimation
                                Jean Luc Apprentice

                                If you spent about 1000 hours of development by now and expect 3000 hours more but do not spend at least several hours profiling the application, especially that you have a performance problem, then you're not doing what you should do.


                                You ought to MEASURE what is happening then optimize accordingly. Just wondering what could go wrong is closer to philosophizing and will not lead you to a solution.


                                If you're not familiar with profiling, invest the time to learn it.

                                • 13. Re: Seam scalability estimation
                                  Todd Morrison Newbie

                                  Step 1: Install profiler


                                  I would suggest you do JVM Profiling on the server while load is active.


                                  http://www.newrelic.com/ has a free service for just this.


                                  This tool can give you Query, Seam, and JVM performance metrics.



                                  Step 2: Get some load


                                  You have a few options:
                                  (a) Let live usage, through Production or QA Server, dictate what is profiled.


                                  Very effective if you have a QA Staff with repeated workflows across iterations.


                                  (b) Using a web recording suite, like Selenium, to replicate a workflow beyond the 'normal' bounds of available usage.


                                  Or, record a use-case and hammer the server with it to exaggerate problems.





                                  These two steps are really more about Testing Theory and JVM development, more than seam.


                                  This is good, since the JVM and Web Development are older than seam, and have been testable for years.



                                  Hope this helps.