1 2 3 4 Previous Next 55 Replies Latest reply on Sep 3, 2012 11:05 AM by fivalo

    Passivating EJB 3 Stateful Session Beans stops the World!

    andy.miller

      Carlo and Andrew,

      I have been running the same load test that I ran for my EAP 4.x performance tuning presentation (uses the EJB 3 Stress Test application we use in QE), and I have discovered what appears to be a pretty major problem. When I first kick-off a run, everything is running quite smoothly and I am actually getting slightly higher results than with EAP 4.2. Then everything just stops. The test just seems to be hung for anywhere between 30 seconds and a minute. Then everything starts again, and it runs like it was running before. In trying to correlate what was happening I found something quite interesting.

      In looking at the JMX console for my stateful session bean, I can see that when everything stops, is when the container is passivating bean instances. In fact, when everything stops, that is the only time that the passivation counts increments in the JMX console. On EAP 4.2/4.3 I noticed similar behavior, but never correlated it to anything, because it would stop only for 5 to 10 seconds, never this extended period of time. Even with these hangs, we are only about 8% slower than before, and if these hangs were eliminated, we would be faster by about 3%. Of course, I am using OpenJDK 6, versus using the Sun JDK 5, and the improvement may be just because of the JVM, but even if that is the case, we are probably a wash in terms of performance.

      In any case, this is a serious issue that needs investigation. Here is some more relevant information:

      The default cache configuration is being used, which should be the StatefulTreeCache, since I have the @Clustered annotation on my stateful bean.

      I can probably reduce the code down to just the servlet and stateful session bean, and strip out the database access, and just put a sleep in the method implementation.

        • 1. Re: Passivating EJB 3 Stateful Session Beans stops the World
          andy.miller

          Some additional information:

          I tested changing the cache maxSize parameter with ejb3-interceptors-aop.xml, and discovered that I am actually using the SimpleStatefulCache. I'm not really sure why. Perhaps something I need to have on the @Clustered annotation?

          Right now, even though I am using a clustered configuration of AS 5 CR2, I don't have another node joining the cluster, but I wouldn't expect the cache to be different based on that.

          • 2. Re: Passivating EJB 3 Stateful Session Beans stops the World
            brian.stansberry

            I'll have a look at how the choice of cache implementation is working in the absence of an explicit @CacheConfig annotation. I'm assuming there's no @CacheConfig annotation; please correct me if I'm wrong.

            • 3. Re: Passivating EJB 3 Stateful Session Beans stops the World
              alrubinger

               

              "andy.miller@jboss.com" wrote:
              I can probably reduce the code down to just the servlet and stateful session bean, and strip out the database access, and just put a sleep in the method implementation.


              In the end we're going to need a more isolated case that may involve invocations on an SFSB alone, or preferably something to test Cache passivation directly. Then we may apply the same test to any of our cache implementations.

              S,
              ALR



              • 4. Re: Passivating EJB 3 Stateful Session Beans stops the World
                alrubinger

                 

                "andy.miller@jboss.com" wrote:
                I tested changing the cache maxSize parameter with ejb3-interceptors-aop.xml, and discovered that I am actually using the SimpleStatefulCache. I'm not really sure why. Perhaps something I need to have on the @Clustered annotation?


                This shouldn't be. We add the proper default annotations if not specified in AOP (ejb3-interceptors-aop.xml):

                <!-- Clustered cache configuration -->
                 <annotation expr="!class(@org.jboss.ejb3.annotation.Cache) AND class(@org.jboss.ejb3.annotation.Clustered)">
                 @org.jboss.ejb3.annotation.Cache ("StatefulTreeCache")
                 </annotation>
                 <annotation expr="!class(@org.jboss.ejb3.annotation.CacheConfig) AND class(@org.jboss.ejb3.annotation.Clustered)">
                 @org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)
                 </annotation>


                S,
                ALR

                • 5. Re: Passivating EJB 3 Stateful Session Beans stops the World
                  alrubinger

                   

                  "andy.miller@jboss.com" wrote:
                  I tested changing the cache maxSize parameter with ejb3-interceptors-aop.xml, and discovered that I am actually using the SimpleStatefulCache.


                  Something I noted w/ SimpleStatefulCache:

                  Without seeing this firsthand, my educated guess is in SessionTimeoutTask.run() where we synchronize on the entire SimpleStatefulCache.cacheMap to do passivation; this will queue up all other Threads looking to create or get a session as they wait to acquire the lock.

                  http://anonsvn.jboss.org/repos/jbossas/projects/ejb3/trunk/core/src/main/java/org/jboss/ejb3/cache/simple/SimpleStatefulCache.java

                  S,
                  ALR




                  • 6. Re: Passivating EJB 3 Stateful Session Beans stops the World
                    andy.miller

                     

                    "bstansberry@jboss.com" wrote:
                    I'll have a look at how the choice of cache implementation is working in the absence of an explicit @CacheConfig annotation. I'm assuming there's no @CacheConfig annotation; please correct me if I'm wrong.


                    Yes, you are correct, there is not @CacheConfig annotation. Only the @Clustered annotation.

                    • 7. Re: Passivating EJB 3 Stateful Session Beans stops the World
                      andy.miller

                       

                      "ALRubinger" wrote:
                      "andy.miller@jboss.com" wrote:
                      I tested changing the cache maxSize parameter with ejb3-interceptors-aop.xml, and discovered that I am actually using the SimpleStatefulCache. I'm not really sure why. Perhaps something I need to have on the @Clustered annotation?


                      This shouldn't be. We add the proper default annotations if not specified in AOP (ejb3-interceptors-aop.xml):

                      <!-- Clustered cache configuration -->
                       <annotation expr="!class(@org.jboss.ejb3.annotation.Cache) AND class(@org.jboss.ejb3.annotation.Clustered)">
                       @org.jboss.ejb3.annotation.Cache ("StatefulTreeCache")
                       </annotation>
                       <annotation expr="!class(@org.jboss.ejb3.annotation.CacheConfig) AND class(@org.jboss.ejb3.annotation.Clustered)">
                       @org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)
                       </annotation>


                      S,
                      ALR


                      I do have what you list above in my configuration. Just before it is also the non-clustered configuration, and that is what I'm picking up, based on my experimentation.

                      I think Brian will discover a problem with the way this is being selected when there is no @CacheConfig annotation.

                      • 8. Re: Passivating EJB 3 Stateful Session Beans stops the World
                        andy.miller

                         

                        "ALRubinger" wrote:
                        "andy.miller@jboss.com" wrote:
                        I can probably reduce the code down to just the servlet and stateful session bean, and strip out the database access, and just put a sleep in the method implementation.


                        In the end we're going to need a more isolated case that may involve invocations on an SFSB alone, or preferably something to test Cache passivation directly. Then we may apply the same test to any of our cache implementations.

                        S,
                        ALR



                        I will work on cutting out just the code for the Stateful Session Bean, so it can be turned into an isolated test case tomorrow.

                        • 9. Re: Passivating EJB 3 Stateful Session Beans stops the World
                          brian.stansberry

                          Andy,

                          Can you post the class declaration for the troublesome bean (along w/ any class annotations) plus any ejb-jar.xml or jboss.xml that impacts the bean?

                          Looking at this, how it determines the cache implementation to use looks pretty straightforward. I misspoke before; it's driven by the org.jboss.ejb3.annotation.Cache annotation, not @CacheConfig.

                          • 10. Re: Passivating EJB 3 Stateful Session Beans stops the World
                            andy.miller

                            Brian,

                            Sure thing. I don't have any ejb-jar.xml or jboss.xml files at all. I tried to use all the defaults as well (convention over configuration) for everything that I could. So here is the class declaration:

                            package services.ejb;
                            
                            import java.io.Serializable;
                            import java.util.ArrayList;
                            import java.util.List;
                            
                            import javax.ejb.Remove;
                            import javax.ejb.Stateful;
                            import javax.persistence.EntityManager;
                            import javax.persistence.PersistenceContext;
                            
                            import org.jboss.annotation.ejb.Clustered;
                            import services.entities.Order;
                            
                            @Clustered
                            @Stateful
                            public class OrderInquiryBean implements OrderInquiry, Serializable {
                            


                            Here is the interface as well:

                            package services.ejb;
                            
                            import java.util.List;
                            import services.entities.Order;
                            
                            public interface OrderInquiry {
                            
                             public int getPaginationForInquiries();
                            
                             public List<Order> findOrdersByCustomer(long customerId);
                            
                             public void cancelSession();
                            
                            }
                            


                            Like I said. I kept everything as simple as possible.

                            • 11. Re: Passivating EJB 3 Stateful Session Beans stops the World
                              andy.miller

                               

                              "bstansberry@jboss.com" wrote:
                              Andy,

                              I misspoke before; it's driven by the org.jboss.ejb3.annotation.Cache annotation, not @CacheConfig.


                              Actually, you were correct according to the documentation. The @Cache annotation is for caching entities, and the @CacheConfig is for StatefulSessionBeans, like what I'm doing.

                              • 12. Re: Passivating EJB 3 Stateful Session Beans stops the World
                                brian.stansberry

                                I believe you are thinking of org.hibernate.annotations.Cache, which is used for entities. The org.jboss.ejb3.annotation.Cache annotation is new and is used to drive what SFSB cache implementation to use. As Andrew's snippet shows, the presence of @Clustered should drive the addition of @org.jboss.ejb3.annotation.Cache ("StatefulTreeCache") which in turn should result in StatefulTreeCache being used.

                                But apparently it doesn't. :)

                                Thanks for the config details; that's what I guessed you were doing, but now I know. Hopefully if I can reproduce.

                                • 13. Re: Passivating EJB 3 Stateful Session Beans stops the World
                                  alrubinger

                                  Andy:

                                  And the test client?

                                  S,
                                  ALR

                                  • 14. Re: Passivating EJB 3 Stateful Session Beans stops the World
                                    andy.miller

                                     

                                    "ALRubinger" wrote:
                                    Andy:

                                    And the test client?

                                    S,
                                    ALR


                                    I had to work on FY'10 budget stuff until late last night, and I had to sit on a call from 6:30 this morning to go over it with a bunch of folks, so I'm just now getting around to creating a slimmed down test case.

                                    1 2 3 4 Previous Next