1 2 3 4 Previous Next 55 Replies Latest reply on Sep 3, 2012 11:05 AM by fivalo Go to original post
      • 15. Re: Passivating EJB 3 Stateful Session Beans stops the World
        brian.stansberry

        I see it. You are using :

        import org.jboss.annotation.ejb.Clustered;


        which was removed a long time ago as part of the reorganization of EJB3 annotations; now it is:

        import org.jboss.ejb3.annotation.Clustered;


        • 16. Re: Passivating EJB 3 Stateful Session Beans stops the World
          andy.miller

           

          "bstansberry@jboss.com" wrote:
          I see it. You are using :

          import org.jboss.annotation.ejb.Clustered;


          which was removed a long time ago as part of the reorganization of EJB3 annotations; now it is:

          import org.jboss.ejb3.annotation.Clustered;


          Okay, so I have to rebuild this against the AS 5 CR 2 libraries then. I was hoping to just be able to take what I had built before, but that's okay. I can add the AS 5 CR 2 libraries to my project manually, and rebuild.

          • 17. Re: Passivating EJB 3 Stateful Session Beans stops the World
            alrubinger

             

            "andy.miller@jboss.com" wrote:
            I had to work on FY'10 budget stuff until late last night, and I had to sit on a call from 6:30 this morning to go over it with a bunch of folks, so I'm just now getting around to creating a slimmed down test case.


            Wasn't meant to pressure you, just showing what I'll need to reflect this in the community suite. :)

            S,
            ALR

            • 18. Re: Passivating EJB 3 Stateful Session Beans stops the World
              wolfc

              That takes care of the cache configuration, but I also want to know what's behind this passivation issue.

              Is passivating or activating the culprit?
              On what OS is this?

              If it was the sync on cacheMap I would say a deadlock would happen, not some timeout of 30 seconds.

              • 19. Re: Passivating EJB 3 Stateful Session Beans stops the World
                brian.stansberry

                What I love about this Andy is it appears you are demonstrating taht what I expect should work does work -- a bean annotated @Clustered w/o a remote interface:

                https://jira.jboss.org/jira/browse/EJBTHREE-1540


                Andrew, remember or chat about this a couple months back?

                • 20. Re: Passivating EJB 3 Stateful Session Beans stops the World
                  brian.stansberry

                   

                  "wolfc" wrote:
                  That takes care of the cache configuration, but I also want to know what's behind this passivation issue.

                  Is passivating or activating the culprit?
                  On what OS is this?

                  If it was the sync on cacheMap I would say a deadlock would happen, not some timeout of 30 seconds.


                  Needs to be proven of course, but SimpleStatefulCache.SessionTimeoutTask holds a lock on the cacheMap while it iterates through the contents and passivates. If there are a lot of beans (knowing Andy, this is a load test and there are) that could take a while. Any SimpleStatefulCache.get() request will block in the meantime.

                  • 21. Re: Passivating EJB 3 Stateful Session Beans stops the World
                    alrubinger

                     

                    "wolfc" wrote:
                    If it was the sync on cacheMap I would say a deadlock would happen, not some timeout of 30 seconds.


                    There's no contention issue. Other Threads will block.

                    Bet you a Bacardi 'n Coke.

                    S,
                    ALR

                    • 22. Re: Passivating EJB 3 Stateful Session Beans stops the World
                      andy.miller

                       

                      "wolfc" wrote:
                      That takes care of the cache configuration, but I also want to know what's behind this passivation issue.

                      Is passivating or activating the culprit?
                      On what OS is this?

                      If it was the sync on cacheMap I would say a deadlock would happen, not some timeout of 30 seconds.


                      The issue is clearly passivation. The passivation count is zero on the first occurrence, and everything hangs, and then the passivation count is non-zero. The only time the hang occurs is when passivation happens. My typical response times on calling the stateful session bean and returning with the result set are between 7 and 40ms.

                      This is being run on Fedora 9, with our OpenJDK 6 JVM.

                      After rebuilding with the AS 5 CR 2 libraries, the problem does go away, because I now have confirmed that I'm using the StatefulTreeCache, which is the default in the ejb3-interceptors-aop.xml based on the fact that I only have the @Clustered annotation:

                      <!-- Clustered cache configuration -->
                       <annotation expr="!class(@org.jboss.ejb3.annotation.Cache) AND class(@org.jboss.ejb3.annotation.Clustered)">
                       @org.jboss.ejb3.annotation.Cache ("StatefulTreeCache")
                       </annotation>
                       <annotation expr="!class(@org.jboss.ejb3.annotation.CacheConfig) AND class(@org.jboss.ejb3.annotation.Clustered)">
                       @org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)
                       </annotation>
                      


                      Clearly the synchronization in the SimpleStatefulCache is a real problem. If I weren't running in a clustered configuration (and didn't have the issue I had because I didn't rebuild with the AS 5 libraries), I would still have the same problem.

                      • 23. Re: Passivating EJB 3 Stateful Session Beans stops the World
                        andy.miller

                         

                        "ALRubinger" wrote:
                        "wolfc" wrote:
                        If it was the sync on cacheMap I would say a deadlock would happen, not some timeout of 30 seconds.


                        There's no contention issue. Other Threads will block.

                        Bet you a Bacardi 'n Coke.

                        S,
                        ALR


                        It's not really a timeout. Its just everything waiting for the passivation to complete. Once it completes, everything starts running again. When I get the test case, I think you will see what I mean. I believe the code you already identified is the real culprit.

                        • 24. Re: Passivating EJB 3 Stateful Session Beans stops the World
                          andy.miller

                           

                          "bstansberry@jboss.com" wrote:
                          I see it. You are using :

                          import org.jboss.annotation.ejb.Clustered;


                          which was removed a long time ago as part of the reorganization of EJB3 annotations; now it is:

                          import org.jboss.ejb3.annotation.Clustered;


                          Okay, I rebuilt the application with the AS 5 libraries, and I am no longer getting the SimpleStatefulCache, but instead am getting the StatefulTreeCache.

                          You cannot configure the StatefulTreeCache (at least I can't see how), so my throughput has now dropped, even though I don't have the hang situation anymore. The StatefulTreeCache doesn't appear to have a maxSize, and it seems to be limited to 10,000 beans.

                          So, the question I have, is how do I get it to pick up the EJB3SFSBSClusteredCache. It appears I would need to add an annotation (@Cache or @CacheConfig), but the documentation for both of these is out-of-date, and the source has no explanation (no javadoc).

                          Any ideas?

                          • 25. Re: Passivating EJB 3 Stateful Session Beans stops the World
                            brian.stansberry

                            You would add an @CacheConfig annotation:

                            @org.jboss.ejb3.annotation.CacheConfig (name="sfsb-cache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)


                            The maxSize param controls the max number of beans.

                            If you don't configure that, and your bean is @Clustered, per the deploy/ejb3-interceptors-aop.xml I have in my workspace, the following should be added to your bean:

                            @org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)


                            It already has a maxSize of 100,000 so I'm not sure how you're ending up with only 10,000. I'll poke around.

                            The name="jboss.cache:service=EJB3SFSBClusteredCache" bit needs to be fixed: https://jira.jboss.org/jira/browse/JBAS-6128 . The way StatefulTreeCache uses that value in AS 5 is to drive a lookup in the CacheManager service, deployed via deploy/cluster/jboss-cache-manager.sar. CacheManager is an on-demand factory for JBoss Cache instances. It's maintains a registry of named JBC configurations, and provides caches based on the name.

                            The name here should be "sfsb-cache". The cache manager is configured to understand the old 4.x name "jboss.cache:service=EJB3SFSBClusteredCache" to be an alias for "sfsb-cache". That allows pre-existing code to work.

                            If you actually wanted to play around with the "sfsb-cache" JBC configuration, you could edit deploy/cluster/jboss-cache-manager.sar/META-INF/jboss-cache-configs.xml. But I don't think that's what you are looking for; you don't control things like the max # of beans via JBC configuration; it's via @CacheConfig.

                            Obviously this needs documenting. :(

                            Does the deploy/ejb3-interceptors-aop.xml file on your JBoss install include this?

                            <annotation expr="!class(@org.jboss.ejb3.annotation.CacheConfig) AND class(@org.jboss.ejb3.annotation.Clustered)">
                             @org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)
                             </annotation>
                            


                            If your maxSize value were something different, that would explain it.

                            • 26. Re: Passivating EJB 3 Stateful Session Beans stops the World
                              brian.stansberry

                              I just moved the above mentioned JIRA to ejb3: https://jira.jboss.org/jira/browse/EJBTHREE-1541

                              • 27. Re: Passivating EJB 3 Stateful Session Beans stops the World
                                andy.miller

                                 

                                "bstansberry@jboss.com" wrote:
                                You would add an @CacheConfig annotation:

                                @org.jboss.ejb3.annotation.CacheConfig (name="sfsb-cache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)


                                The maxSize param controls the max number of beans.

                                If you don't configure that, and your bean is @Clustered, per the deploy/ejb3-interceptors-aop.xml I have in my workspace, the following should be added to your bean:

                                @org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)


                                It already has a maxSize of 100,000 so I'm not sure how you're ending up with only 10,000. I'll poke around.

                                The name="jboss.cache:service=EJB3SFSBClusteredCache" bit needs to be fixed: https://jira.jboss.org/jira/browse/JBAS-6128 . The way StatefulTreeCache uses that value in AS 5 is to drive a lookup in the CacheManager service, deployed via deploy/cluster/jboss-cache-manager.sar. CacheManager is an on-demand factory for JBoss Cache instances. It's maintains a registry of named JBC configurations, and provides caches based on the name.

                                The name here should be "sfsb-cache". The cache manager is configured to understand the old 4.x name "jboss.cache:service=EJB3SFSBClusteredCache" to be an alias for "sfsb-cache". That allows pre-existing code to work.

                                If you actually wanted to play around with the "sfsb-cache" JBC configuration, you could edit deploy/cluster/jboss-cache-manager.sar/META-INF/jboss-cache-configs.xml. But I don't think that's what you are looking for; you don't control things like the max # of beans via JBC configuration; it's via @CacheConfig.

                                Obviously this needs documenting. :(

                                Does the deploy/ejb3-interceptors-aop.xml file on your JBoss install include this?

                                <annotation expr="!class(@org.jboss.ejb3.annotation.CacheConfig) AND class(@org.jboss.ejb3.annotation.Clustered)">
                                 @org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)
                                 </annotation>
                                


                                If your maxSize value were something different, that would explain it.


                                I do have the above in my ejb3-interceptors-aop.xml.

                                Here is the behavior I see:

                                With just @Clustered and no Cache or CacheConfig annotation, I get the StatefulTreeCache.

                                I think you say that using jboss.cache:service=EJB3SFSBClusteredCache doesn't work, and you would be correct. I had that, and I get no deployment error, but I still get StatefulTreeCache.

                                With @Clustered and @CacheConfig(name="sfsb-cache", maxSize=125000) I still get StatefulTreeCache (not the EJB3SFSBStatefulTreeCache), and the size shows -1 in the jmx-console, and when I run it I still only get 10,000 as the max size. I just figured 10,000 because it never goes above that when refreshing, and when it gets close to that size is when passivation occurs.

                                This just doesn't appear to be working correctly at all.

                                • 28. Re: Passivating EJB 3 Stateful Session Beans stops the World
                                  brian.stansberry

                                   

                                  "andy.miller@jboss.com" wrote:

                                  I do have the above in my ejb3-interceptors-aop.xml.

                                  Here is the behavior I see:

                                  With just @Clustered and no Cache or CacheConfig annotation, I get the StatefulTreeCache.


                                  Good.

                                  I think you say that using jboss.cache:service=EJB3SFSBClusteredCache doesn't work, and you would be correct. I had that, and I get no deployment error, but I still get StatefulTreeCache.


                                  No, that's not what I meant; what you see is what I expect.

                                  There's a couple of concepts here; one of which users normally don't need to worry about, the other they do.

                                  The first is what implementation of the "StatefulCache" interface EJB3 will use. This is the one users normally don't have to worry about. That interface defines the SPI the EJB3 container uses to interact with the caching subsystem. "StatefulTreeCache" is an implementation of that SPI that knows how to interact with JBC. "SimpleStatefulCache" is an implementation that works with the filesystem.

                                  You only became aware of this stuff because you had the @Clustered package issue that caused EJB3 to use the wrong one. With that fixed, with @Clustered you should always see "StatefulTreeCache".

                                  The second concept is how to configure whatever impl of StatefulCache is used. That's an end user concern and is done via the @CacheConfig annotation. The ejb3-interceptors-aop.xml stuff you saw is just a way of setting defaults if you don't add the annotation yourself.

                                  StatefulTreeCache uses the @CacheConfig.name attribute to tell it what JBoss Cache configuration to use. It gets the JBC instance from the CacheManager service by passing in the @CacheConfig.name value. The standard value it should pass in is "sfsb-cache", but if it passes in the legacy AS 4.x default value of "jboss.cache:service=EJB3SFSBClusteredCache", the CacheManager knows that's just an alias for the AS 5.x default of "sfsb-cache".

                                  So, this part sounds like its working just fine.

                                  With @Clustered and @CacheConfig(name="sfsb-cache", maxSize=125000) I still get StatefulTreeCache (not the EJB3SFSBStatefulTreeCache), and the size shows -1 in the jmx-console, and when I run it I still only get 10,000 as the max size. I just figured 10,000 because it never goes above that when refreshing, and when it gets close to that size is when passivation occurs.


                                  Yes, why the config value of maxSize=100,000 doesn't seem to be respected is what I need to figure out.

                                  • 29. Re: Passivating EJB 3 Stateful Session Beans stops the World
                                    andy.miller

                                     

                                    "bstansberry@jboss.com" wrote:
                                    "andy.miller@jboss.com" wrote:

                                    I do have the above in my ejb3-interceptors-aop.xml.

                                    Here is the behavior I see:

                                    With just @Clustered and no Cache or CacheConfig annotation, I get the StatefulTreeCache.


                                    Good.

                                    I think you say that using jboss.cache:service=EJB3SFSBClusteredCache doesn't work, and you would be correct. I had that, and I get no deployment error, but I still get StatefulTreeCache.


                                    No, that's not what I meant; what you see is what I expect.

                                    There's a couple of concepts here; one of which users normally don't need to worry about, the other they do.

                                    The first is what implementation of the "StatefulCache" interface EJB3 will use. This is the one users normally don't have to worry about. That interface defines the SPI the EJB3 container uses to interact with the caching subsystem. "StatefulTreeCache" is an implementation of that SPI that knows how to interact with JBC. "SimpleStatefulCache" is an implementation that works with the filesystem.

                                    You only became aware of this stuff because you had the @Clustered package issue that caused EJB3 to use the wrong one. With that fixed, with @Clustered you should always see "StatefulTreeCache".

                                    The second concept is how to configure whatever impl of StatefulCache is used. That's an end user concern and is done via the @CacheConfig annotation. The ejb3-interceptors-aop.xml stuff you saw is just a way of setting defaults if you don't add the annotation yourself.

                                    StatefulTreeCache uses the @CacheConfig.name attribute to tell it what JBoss Cache configuration to use. It gets the JBC instance from the CacheManager service by passing in the @CacheConfig.name value. The standard value it should pass in is "sfsb-cache", but if it passes in the legacy AS 4.x default value of "jboss.cache:service=EJB3SFSBClusteredCache", the CacheManager knows that's just an alias for the AS 5.x default of "sfsb-cache".

                                    So, this part sounds like its working just fine.

                                    With @Clustered and @CacheConfig(name="sfsb-cache", maxSize=125000) I still get StatefulTreeCache (not the EJB3SFSBStatefulTreeCache), and the size shows -1 in the jmx-console, and when I run it I still only get 10,000 as the max size. I just figured 10,000 because it never goes above that when refreshing, and when it gets close to that size is when passivation occurs.


                                    Yes, why the config value of maxSize=100,000 doesn't seem to be respected is what I need to figure out.


                                    Thanks for explaining. I was misunderstanding the legacy part.

                                    One final question. If I remove the @CacheConfig annotation, can I add the maxSize and the other parameters to the ejb3-interceptors-aop.xml, like it has for the other two entries?