WF 12: Memory leak using Infinispan as SFSB cache?

[lonny27]

Hallo!

WildFly 12.0.0.Final can use Infinispan (9.1.6.Final) as SFSB cache. Doing so, I observed a strange behavior (which can be the result of a misinterpretation of what I observed, an error in my WildFly configuration, my projects nature (an old Seam 2.2 application) or a bug in WildFlys Infinispan integration), where org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupEntrys are not removed from the cache, resulting in a memory leak.

Observation

For each SFSB, two Infinispan cache entries with the same org.jboss.ejb.client.UUIDSessionID are created

• a org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupEntry
• and a org.wildfly.clustering.ejb.infinispan.bean.InfinispanBeanEntry

If a SFSB is destroyed, it's corresponding InfinispanBeanEntry is removed from the Infinispan cache, but the corresponding InfinispanBeanGroupEntry is not removed. The cache statistics changes as following

• numberOfEntries: from 2 to 1
• removeHits: from 0 to 1
• removeMisses: from 0 to 1

Conclusion

If a SFSB is destroyed, both corresponding cache entries should be removed from the Infinispan cache. Entries of type org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroup fail to be removed.

Since all entries in the cache are org.infinispan.container.entries.ImmortalCacheEntrys, they will never be removed from the cache until WildFly is restarted (file-store purge="true"), causing a memory leak (well, they will be passivated at some point).

Infinispan subsystem configuration

<cache-container name="ejb" aliases="sfsb" default-cache="passivation" module="org.wildfly.clustering.ejb.infinispan" statistics-enabled="true">

  <local-cache name="passivation" statistics-enabled="true">

    <locking isolation="REPEATABLE_READ"/>

    <transaction mode="BATCH"/>

    <file-store passivation="true" purge="true"/>

  </local-cache>

</cache-container>

Attachment

Infinispan trace log when a InfinispanBeanEntry of a SFSB is removed from the cache.

I found a workaround by removing the stray InfinispanBeanGroups manually, but that seems to be more a hack than a solution.

I would appreciate any insight/help on this subject!

Kind Regards

Ronald

[pferraro]

Does your @Stateful EJB contain any references to other @Stateful EJBs?  If so, this implies the outer bean was removed, but the nested bean was not (removes don't automatically cascade).  If not, this is probably a bug.

Can you describe your SFSB access?  Are these accessed remotely? or locally?  How are the your SFSB removes initiated?  @Remove method?  @StatefulTimeout? Are they contextual (i.e. scoped)?

[lonny27]

Hi Paul! I was kinda hoping that this post would catch your attention :-)

The project this happens uses Seam 2.2.2.Final.

For testing I used an "empty" SFSB, that does not reference anything else. The SFSB access is local and its scope is @org.jboss.seam.annotations.Scope(ScopeType.CONVERSATION)

The SFSB creation is triggered by Seam during the JSF page render phase on the first hit of the SFSB Seam component name in the page (afterwards the cache contains only two entries, the BeanGroupEntry and the BeanEntry).

The SFSB removal is also triggered by Seam - which invokes the @java.ejb.Remove annotated method of the SFSB (using additionally Seams own org.jboss.seam.annotations.Destroy annotation makes no difference) - during logout, when the Seam context is destroyed (the BeanGroupEntry survives and its totalUsage value is changed from 0 to 1).

I tried to debug what causes the removeMisses entry in the statistics, but hit soon a dead end. Infinispan is not easy to debug. I would need to get really familiar with its source code. Maybe you can point me in the right direction.