14 Replies Latest reply on Oct 14, 2008 10:05 PM by pmuir

    Application instability in multiuser environment

      Hi there,


      after a lot of LOCs, coffee and blood our QM department tested my application. It runs on JBoss AS 4.2.2.GA, uses Seam 2.0.1.GA, Hibernate and ICEfaces 1.7.1.
      The results are not that nice as I expected they would be. The QM department reported errors that I'm not able to reproduce so I think the most of the errors are a result of a general instability of my application when it is used with multiple users.


      So I want to ask if somebody could explain me how to improve the stability and performance of my application in mutliuser environments. Are there any general articles or books on that topic available? I mostly use the session scope for backing beans; is that okay or not? Is ICEfaces too heavy?
      Some of the reports say that there are exceptions with messages like Could not find stateful bean: .... How can I avoid those situations? Or how do they occur?


      Thanks in advance
      Jens

        • 1. Re: Application instability in multiuser environment
          gjeudy

          We would be glad to help you but I think this has to be taken case by case. If you provide detailed information about the problems perhaps we can help. I'm sure you get dig out the logs and give us some stacktraces.

          • 2. Re: Application instability in multiuser environment

            I hoped I could get some general information about stability. Of course I have the log file, but it's 3.7GB big. It's not really comfortable getting some interesting data out of this file.


            I think we will examine this behaviour later but for now we would like to have an idea what could be the problem.


            BTW: How can I avoid having logged everything. There are a lot of DEBUG logs.

            • 3. Re: Application instability in multiuser environment
              gjeudy

              In my experience there is no silver-bullet solution on stability. You have to investigate the specific issues you are having and remediate. There is so many things at any layers that could cause instability.


              First off you should review your logging solution to make your log file more manageable. Check out your conf/jboss-log4j.xml. You might want to add:


              <category name="org.jboss.seam">
                        <priority value="INFO"/>
                   </category>
                   <category name="org.ajax4jsf">
                        <priority value="INFO"/>
                   </category>
                   <category name="org.springframework">
                        <priority value="INFO"/>
                   </category>
                   <category name="org.hibernate">
                        <priority value="INFO"/>
                   </category>
                   <category name="org.hibernate.type">
                        <priority value="TRACE" class="org.jboss.logging.XLevel"/>
                   </category>
                   <category name="org.hibernate.SQL">
                        <priority value="DEBUG"/>
                   </category>  
              



              Generally you don't want to log lower than INFO unless you need all the info because you are debugging in a specific library. The hibernate DEBUG and TRACE settings above will get you all the SQL statements fired at DB along with parameter values.


              if you are running under a *nix system you can use split command to split a big log file in chunks.


              The other thing you can do is route specific log entries from java packages to different appenders so you don't have everything in the same file, however it does help to have everything in the same file to figure out what is going on across threads.


              I'm not a log4j expert but it's probably possible to add the thread-name in each log entries. This would help you troubleshoot concurrency related issues.

              • 4. Re: Application instability in multiuser environment
                fernando_jmt

                If I recall correctly, I see a lot of Could not find stateful bean: ... exceptions registered in the old JBoss Seam forum. Even I got some of them in my earlier times.


                This is related to EJB Stateful Session Beans (SFSB) activation and passivation.


                When Bean instance is not used for a while by client then EJB Container removes it from memory and puts it in secondary storage so that the memory can be reused. This is called Passivation.


                When Client calls the bean instance again then Container takes the passivated bean from secondary storage and puts it in memory to serve the client request. This is called Activation.


                The Stateful Session beans are not in a pasivated state for ever, they timeout (instance is completely removed from the secondary storage).


                So what is happening seems to be the following:



                • An SFSB is initialized within a conversational component. Suppose your conversation is configured to timeout in 5 minutes. Such a component get a reference to such SFSB.

                • The SFSB is pasivated by the container.

                • The SFSB timeout in 2 minute (a sample, this is dependant of the Application server defaults and configuration).

                • Then the conversational component tries to access to the SFSB, but it does not exist anymore (it timed out), so you will get a nicest exception (Could not find stateful bean: ...).



                That could be what is happening in your application.


                Months ago I saw that Gavin King registered an issue in the JBoss EJB3 forum asking to them to make the default timeout of EJBs greater than the HTTP Session configured timeout, I don't remember the issue number and I don't know if it's resolved in the JBoss 5. Maybe you can look for it.



                HTH.




                • 5. Re: Application instability in multiuser environment

                  OK. We'll test again with modified log settings. Thank you for that hint.

                  • 6. Re: Application instability in multiuser environment
                    fernando_jmt

                    As far as I know you can avoid such errors having the combination: SFSB timout (JBoss config) = Conversation timeout (Your app) = HttpSession timeout (Your app).


                    • 7. Re: Application instability in multiuser environment
                      brachie

                      Hi,


                      we are having the same problem with our live going Seam application. In the logs I get several messages like this:



                      |2008-10-13 18:04:34,993 WARN  [org.jboss.seam.Component] Exception calling component @Destroy method: viewStateHolder
                      javax.ejb.NoSuchEJBException: Could not find stateful bean: 61h5f1e-qnpm1u-fm6dzg6m-1-fm9aa9my-60l
                              at org.jboss.ejb3.cache.simple.SimpleStatefulCache.get(SimpleStatefulCache.java:390)
                              at org.jboss.ejb3.cache.simple.SimpleStatefulCache.get(SimpleStatefulCache.java:375)
                              at org.jboss.ejb3.stateful.StatefulInstanceInterceptor.invoke(StatefulInstanceInterceptor.java:61)
                              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                              at org.jboss.ejb3.stateful.StatefulRemoveInterceptor.invoke(StatefulRemoveInterceptor.java:97)
                              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                              at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                              at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110)
                              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                              at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                              at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                              at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                              at org.jboss.ejb3.stateful.StatefulContainer.localInvoke(StatefulContainer.java:206)
                              at org.jboss.ejb3.stateful.StatefulLocalProxy.invoke(StatefulLocalProxy.java:119)|
                      



                      You can set the timeout settings


                       idleTimeoutSeconds=7200, removalTimeoutSeconds=0 



                      in the /server/default/deploy/ejb3-interceptors-aop.xml file of your JBoss installation like this:



                       <domain name="Stateful Bean" extends="Base Stateful Bean" inheritBindings="true">
                            <!-- NON Clustered cache configuration -->
                            <annotation expr="!class(@org.jboss.annotation.ejb.cache.Cache) AND !class(@org.jboss.annotation.ejb.Clustered)">
                               @org.jboss.annotation.ejb.cache.Cache (org.jboss.ejb3.cache.simple.SimpleStatefulCache.class)
                            </annotation>
                            <annotation expr="!class(@org.jboss.annotation.ejb.cache.simple.PersistenceManager) AND !class(@org.jboss.annotation.ejb.Clustered)">
                               @org.jboss.annotation.ejb.cache.simple.PersistenceManager (org.jboss.ejb3.cache.simple.StatefulSessionFilePersistenceManager.class)
                            </annotation>
                            <annotation expr="!class(@org.jboss.annotation.ejb.cache.simple.CacheConfig) AND !class(@org.jboss.annotation.ejb.Clustered)">
                               @org.jboss.annotation.ejb.cache.simple.CacheConfig (maxSize=100000, idleTimeoutSeconds=7200, removalTimeoutSeconds=0)
                            </annotation>
                      ...



                      where idleTimeoutSeconds is the time in seconds after which the SFSB passivates and removalTimeoutSeconds is the timeout after which the complete SFSB is destroyed (0 means no timeout)(see here: https://www.jboss.org/community/docs/DOC-9628).


                      So in my config the timeout for passivation is 7200 seconds (2 hours). The problem is that I still get the error in my logs.


                      If my understanding is correct the SFSB is not destroyed after all (all it does is passivating after 2 hours!) because the removalTimeoutSeconds is set to infinite. So why do I get the errors assuming that my conversation is sill active? How can that be?


                      Any help is appreciated!


                      Alexander

                      • 8. Re: Application instability in multiuser environment
                        pmuir

                        related is https://jira.jboss.org/jira/browse/JBAS-5778. Fernando, we need to find that EJBTHREE issue - can you trace it down?

                        • 9. Re: Application instability in multiuser environment
                          pmuir

                          We are going to work on clustering for Seam 2.1.1 in tandem with Brain and co. from the JBoss Clustering team - so please, if you know of blogs about clustering Seam, or hints and tips for running in a cluster, or pain points, please add them to the knowledge base :-)

                          • 10. Re: Application instability in multiuser environment
                            fernando_jmt

                            Pete Muir wrote on Oct 14, 2008 13:44:


                            related is https://jira.jboss.org/jira/browse/JBAS-5778. Fernando, we need to find that EJBTHREE issue - can you trace it down?


                            I found it: https://jira.jboss.org/jira/browse/EJBTHREE-788

                            • 11. Re: Application instability in multiuser environment
                              obfuscator

                              A few things:



                              1. Don't use injections in multithreaded state (not safe) - use @Create or equiv

                              2. Check logging thread locking - could be significant (very much so!) - use async append or shut down.

                              3. Check 1000x setter calls problem - use lazy init

                              4. Don't set the db isolation levels too high

                              5. Don't use JMS bridging with JBoss MQ (duplicate messages can happen i have noticed), use clustering config (all config) and MDB:s work like charm

                              6. Make sure you handle timeouts gracefully (this you already noticed :))




                              When we discovered the thread locking problem, the app suddenly ran 10x as fast.


                              • 12. Re: Application instability in multiuser environment
                                pmuir

                                Alexander T wrote on Oct 14, 2008 20:50:



                                1. Don't use injections in multithreaded state (not safe) - use @Create or equiv





                                Can you elaborate... What version of Seam?

                                • 13. Re: Application instability in multiuser environment
                                  obfuscator

                                  Well, when injecting variables in application-scoped components I've had race conditions several times, with several contexts nulling and resetting those variables. But now when you mention it, that was mostly earlier on. Has there been a fix for that now? I believe I got that tip on the seam channel about a year ago or so. I'm doing initialization in @Create regularly now, so I haven't tried if it works lately...

                                  • 14. Re: Application instability in multiuser environment
                                    pmuir

                                    Fixed in 2.1.0.CR1 I think (if we are talking about the same issue).