StatefulSessionComponent causing OOM in AS7

[mick_mcgovern1]

Hi

We have a banking app - core transaction engine - that gradually runs out of memory with the above mentioned class appearing to be the culprit.

We have over 1000 different Stateful session bean classes in the app but only a fraction of them are actually instantiated during a run.

Here is a bit of info from the Eclipse profiler that shows the StatefulSessionComponent:

One instance of"org.jboss.as.ejb3.component.stateful.StatefulSessionComponent" loaded by"org.jboss.modules.ModuleClassLoader @ 0x53b3280" occupies 76,054,856 (15.19%) bytes. The memory is accumulated in one instance of"java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system class loader>".

...

cache org.jboss.as.ejb3.cache.impl.backing.NonPassivatingBackingCacheImpl @ 0x116820b8

backingCache org.jboss.as.ejb3.cache.impl.SimpleCache @ 0x11633198

cache org.jboss.as.ejb3.component.stateful.StatefulSessionComponent @ 0x115d5418

.....

The app has plenty of memory (1GB) and we are limiting the number of SFSB in the cache (however the number set for the cache doesn't help  - e.g. setting it to 10)

Any ideas?

Thanks

Michael

[jaikiran]

Which exact version of AS7 and can you post your bean and EJB3 subsystem configurations?

[nickarls]

Also, are you using @EJB or @Inject injections, do you have any @ApplicationScoped/@Singleton stuff (or anything else more long-lived than a session)?

[ctomc]

Also to help identify how the memmory usage is distributed.

could you also post what are the jvm paramteres with that you run AS7 with.

what exact jvm do you use, 32/64bit?

How big is you deployed application.

As what your are saying is that 15.19% of heap memmory consists to 76MB in size which is actualy not that much, and that would make your total heap size around 477MB and that is half of

what you are saying it is allocated for application.

--

tomaz

[mick_mcgovern1]

thanks for the response.

We have many @EJB throughout - though most of them are stateless.

A couple of @Singletons

I suspect we have some scope issues where we are nesting calls and not getting to the remove call.

[mick_mcgovern1]

thanks for the response

we have tried many settings - I think the sample was from a 512 Xmx setting -but regardless of max heap it still runs out - just takes longer the more you give it.

The 76MB represents a single instance of the StatelessSessionComponent - which in itself is no great issue. The profile shows that virtually all the memory is used up by JBoss related classes - our own making only a couple of percent -

[jaikiran]

I'm confused. Your first post talks about StatefulSessionComponent and then the next few posts talk about stateless and singleton. So I can't really say what's wrong. You'll have to provide much more details including configurations in the subsystem, the bean code/configurations and even the profiler snapshots (if possible). And you haven't answered the question about the exact version of AS7 you are using.

[sfcoy]

I'm going to chime in here and mention that you *must* manage the lifecycle of your stateful session beans, otherwise you will get memory leaks. This means ensuring that you invoke their @Remove methods when you're done with them.

[mick_mcgovern1]

Sorry for that (I'm working with remote colleagues)

The version is 7.1.0 but aso tried 7.1.1.

We have a large app with Stateful, stateless, SIngleton etc.  It is the Stateful that are the cause of the problem  and, as pointed out, we need to ensure we do the life-cycle thing right, That is where we are focussing now , Thanks everyone.

[mick_mcgovern1]

thanks - I'm pretty sure somewhere we are not doing that.

[mick_mcgovern1]

Just a question: Does the call to a "@Remove" method cascade removal of other stateful beans held or do they need to be removed explictly? Thanks.

[mick_mcgovern1]

I've done an exercise where I lookup a SFSB 1000 times in a loop - followed directly by a call to the remove method (annotated @Remove)

Running with 1,5GB Xmx I crash the JVM after about 5 requests (5 x 1000 create/removed)

Just for kicks I did a forced GC each request - that made it last a little longer but same issue.

I took some dumps and I see that all my created SFSB are still "in memory". I know there are no guarantees about what a GC call should do but when I'm running out of memeory and I've called remove on my SFSB I dont expect them ALL to still be sitting in memory.

From Eclipse MAT

Class Name                                                                          | Objects | Shallow Heap | Retained Heap
-----------------------------------------------------------------------------------------------------------------------------
com.vantage.capital.tranprocessing.standinginstruction.ApplyStandingInstructionsBean|   4,000 |    1,088,000 |  >= 1,664,008
-----------------------------------------------------------------------------------------------------------------------------

This dump taken after 4 requests (4 x 1000)

Is there something more to SFSB that just having the annotated remove method and calling when I'm finished with it? Or is JBoss not bothering to remove them from some cache: (Again from MAT - outbound path from my SFSB show reference to cache)

Class Name                                                                                                                                     | Shallow Heap | Retained Heap
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
com.vantage.capital.tranprocessing.standinginstruction.ApplyStandingInstructionsBean @ 0xcab01bf8                                              |          272 |           416
'- instance org.jboss.as.naming.ValueManagedReferenceFactory$ValueManagedReference @ 0xcab01be0                                                |           24 |           440
   '- value java.util.concurrent.atomic.AtomicReference @ 0xcab01bc8                                                                           |           24 |           464
      |- instanceReference org.jboss.as.ejb3.component.stateful.StatefulSessionComponentInstance @ 0xcab01b40                                  |          136 |        95,016
      |- referenceReference org.jboss.as.ee.component.ManagedReferenceReleaseInterceptorFactory$ManagedReferenceReleaseInterceptor @ 0xcab602b8|           24 |            24
      |  '- [0] org.jboss.invocation.Interceptor[1] @ 0xcab60298                                                                               |           32 |            56
      |     '- a java.util.Arrays$ArrayList @ 0xcab60278                                                                                       |           32 |            88
      |        '- interceptors org.jboss.invocation.WeavedInterceptor @ 0xcab60260                                                             |           24 |           112
      |           '- [5] org.jboss.invocation.Interceptor[7] @ 0xcab60210                                                                      |           80 |         3,400
      |              '- a java.util.Arrays$ArrayList @ 0xcab601f0                                                                              |           32 |         3,432
      |                 '- interceptors org.jboss.invocation.ChainedInterceptor @ 0xcab601d8                                                   |           24 |         3,456
      |                    '- preDestroy org.jboss.as.ejb3.component.stateful.StatefulSessionComponentInstance @ 0xcab01b40                    |          136 |        95,016
      |                       '- wrapped org.jboss.as.ejb3.cache.impl.backing.NonPassivatingBackingCacheEntry @ 0xcab01b08                     |           56 |        95,144
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Thanks

[sfcoy]

There is no notion of cascading remove operations on stateful session beans. They must be removed explicitly.

[mick_mcgovern1]

Further to this claim that JBoss AS 7 is not removing Stateful session beans - I've added debug in a "PreDestroy" method and it is never getting called - even when I'm running out of memory and the JVM is spending all it's time GCing - the unused/removed SFSBs remain in memory.

[sfcoy]

This would indeed appear to be a bug.

The work around is to add the @Remove annotation to the implementation of the remove method as well.