1 2 Previous Next 26 Replies Latest reply on Sep 21, 2016 2:38 PM by Sidney Zurch

    StatefulSessionComponent causing OOM in AS7

    Michael McGovern Newbie

      Hi

      We have a banking app - core transaction engine - that gradually runs out of memory with the above mentioned class appearing to be the culprit.

       

      We have over 1000 different Stateful session bean classes in the app but only a fraction of them are actually instantiated during a run.

       

      Here is a bit of info from the Eclipse profiler that shows the StatefulSessionComponent:

       

      One instance of"org.jboss.as.ejb3.component.stateful.StatefulSessionComponent" loaded by"org.jboss.modules.ModuleClassLoader @ 0x53b3280" occupies 76,054,856 (15.19%) bytes. The memory is accumulated in one instance of"java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system class loader>".

       

      ...

      cache org.jboss.as.ejb3.cache.impl.backing.NonPassivatingBackingCacheImpl @ 0x116820b8

      backingCache org.jboss.as.ejb3.cache.impl.SimpleCache @ 0x11633198

      cache org.jboss.as.ejb3.component.stateful.StatefulSessionComponent @ 0x115d5418

      .....

       

       

      The app has plenty of memory (1GB) and we are limiting the number of SFSB in the cache (however the number set for the cache doesn't help  - e.g. setting it to 10)

       

      Any ideas?

       

      Thanks

       

      Michael

        • 1. Re: StatefulSessionComponent causing OOM in AS7
          jaikiran pai Master

          Which exact version of AS7 and can you post your bean and EJB3 subsystem configurations?

          • 2. Re: StatefulSessionComponent causing OOM in AS7
            Nicklas Karlsson Master

            Also, are you using @EJB or @Inject injections, do you have any @ApplicationScoped/@Singleton stuff (or anything else more long-lived than a session)?

            • 3. Re: StatefulSessionComponent causing OOM in AS7
              Tomaz Cerar Master

              Also to help identify how the memmory usage is distributed.

              could you also post what are the jvm paramteres with that you run AS7 with.

              what exact jvm do you use, 32/64bit?

              How big is you deployed application.

              As what your are saying is that 15.19% of heap memmory consists to 76MB in size which is actualy not that much, and that would make your total heap size around 477MB and that is half of

              what you are saying it is allocated for application.

               

               

              --

              tomaz

              • 4. Re: StatefulSessionComponent causing OOM in AS7
                Michael McGovern Newbie

                thanks for the response.

                 

                We have many @EJB throughout - though most of them are stateless.

                 

                A couple of @Singletons

                 

                I suspect we have some scope issues where we are nesting calls and not getting to the remove call.

                • 5. Re: StatefulSessionComponent causing OOM in AS7
                  Michael McGovern Newbie

                  thanks for the response

                   

                  we have tried many settings - I think the sample was from a 512 Xmx setting -but regardless of max heap it still runs out - just takes longer the more you give it.

                   

                  The 76MB represents a single instance of the StatelessSessionComponent - which in itself is no great issue. The profile shows that virtually all the memory is used up by JBoss related classes - our own making only a couple of percent -

                  • 6. Re: StatefulSessionComponent causing OOM in AS7
                    jaikiran pai Master

                    I'm confused. Your first post talks about StatefulSessionComponent and then the next few posts talk about stateless and singleton. So I can't really say what's wrong. You'll have to provide much more details including configurations in the subsystem, the bean code/configurations and even the profiler snapshots (if possible). And you haven't answered the question about the exact version of AS7 you are using.

                    • 7. Re: StatefulSessionComponent causing OOM in AS7
                      Stephen Coy Master

                      I'm going to chime in here and mention that you *must* manage the lifecycle of your stateful session beans, otherwise you will get memory leaks. This means ensuring that you invoke their @Remove methods when you're done with them.

                      1 of 1 people found this helpful
                      • 8. Re: StatefulSessionComponent causing OOM in AS7
                        Michael McGovern Newbie

                        Sorry for that (I'm working with remote colleagues)

                         

                        The version is 7.1.0 but aso tried 7.1.1.

                         

                        We have a large app with Stateful, stateless, SIngleton etc.  It is the Stateful that are the cause of the problem  and, as pointed out, we need to ensure we do the life-cycle thing right, That is where we are focussing now , Thanks everyone.

                        • 9. Re: StatefulSessionComponent causing OOM in AS7
                          Michael McGovern Newbie

                          thanks - I'm pretty sure somewhere we are not doing that.

                          • 10. Re: StatefulSessionComponent causing OOM in AS7
                            Michael McGovern Newbie

                            Just a question: Does the call to a "@Remove" method cascade removal of other stateful beans held or do they need to be removed explictly? Thanks.

                            • 11. Re: StatefulSessionComponent causing OOM in AS7
                              Michael McGovern Newbie

                              I've done an exercise where I lookup a SFSB 1000 times in a loop - followed directly by a call to the remove method (annotated @Remove)

                               

                              Running with 1,5GB Xmx I crash the JVM after about 5 requests (5 x 1000 create/removed)

                              Just for kicks I did a forced GC each request - that made it last a little longer but same issue.

                               

                              I took some dumps and I see that all my created SFSB are still "in memory". I know there are no guarantees about what a GC call should do but when I'm running out of memeory and I've called remove on my SFSB I dont expect them ALL to still be sitting in memory.

                               

                              From Eclipse MAT

                               

                              Class Name                                                                          | Objects | Shallow Heap | Retained Heap
                              -----------------------------------------------------------------------------------------------------------------------------
                              com.vantage.capital.tranprocessing.standinginstruction.ApplyStandingInstructionsBean|   4,000 |    1,088,000 |  >= 1,664,008
                              -----------------------------------------------------------------------------------------------------------------------------

                              This dump taken after 4 requests (4 x 1000)

                               

                              Is there something more to SFSB that just having the annotated remove method and calling when I'm finished with it? Or is JBoss not bothering to remove them from some cache: (Again from MAT - outbound path from my SFSB show reference to cache)

                               

                              Class Name                                                                                                                                     | Shallow Heap | Retained Heap
                              ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
                              com.vantage.capital.tranprocessing.standinginstruction.ApplyStandingInstructionsBean @ 0xcab01bf8                                              |          272 |           416
                              '- instance org.jboss.as.naming.ValueManagedReferenceFactory$ValueManagedReference @ 0xcab01be0                                                |           24 |           440
                                 '- value java.util.concurrent.atomic.AtomicReference @ 0xcab01bc8                                                                           |           24 |           464
                                    |- instanceReference org.jboss.as.ejb3.component.stateful.StatefulSessionComponentInstance @ 0xcab01b40                                  |          136 |        95,016
                                    |- referenceReference org.jboss.as.ee.component.ManagedReferenceReleaseInterceptorFactory$ManagedReferenceReleaseInterceptor @ 0xcab602b8|           24 |            24
                                    |  '- [0] org.jboss.invocation.Interceptor[1] @ 0xcab60298                                                                               |           32 |            56
                                    |     '- a java.util.Arrays$ArrayList @ 0xcab60278                                                                                       |           32 |            88
                                    |        '- interceptors org.jboss.invocation.WeavedInterceptor @ 0xcab60260                                                             |           24 |           112
                                    |           '- [5] org.jboss.invocation.Interceptor[7] @ 0xcab60210                                                                      |           80 |         3,400
                                    |              '- a java.util.Arrays$ArrayList @ 0xcab601f0                                                                              |           32 |         3,432
                                    |                 '- interceptors org.jboss.invocation.ChainedInterceptor @ 0xcab601d8                                                   |           24 |         3,456
                                    |                    '- preDestroy org.jboss.as.ejb3.component.stateful.StatefulSessionComponentInstance @ 0xcab01b40                    |          136 |        95,016
                                    |                       '- wrapped org.jboss.as.ejb3.cache.impl.backing.NonPassivatingBackingCacheEntry @ 0xcab01b08                     |           56 |        95,144
                              ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                               

                              Thanks

                               

                               

                               

                               

                               

                               

                              • 12. Re: StatefulSessionComponent causing OOM in AS7
                                Stephen Coy Master

                                There is no notion of cascading remove operations on stateful session beans. They must be removed explicitly.

                                • 13. Re: StatefulSessionComponent causing OOM in AS7
                                  Michael McGovern Newbie

                                  Further to this claim that JBoss AS 7 is not removing Stateful session beans - I've added debug in a "PreDestroy" method and it is never getting called - even when I'm running out of memory and the JVM is spending all it's time GCing - the unused/removed SFSBs remain in memory.

                                  • 14. Re: StatefulSessionComponent causing OOM in AS7
                                    Stephen Coy Master

                                    This would indeed appear to be a bug.

                                     

                                    The work around is to add the @Remove annotation to the implementation of the remove method as well.

                                    1 2 Previous Next