consistently increases .dat file and eventually, OOM facing

[ltarley]

Hi Team,

i am using WF-8.0.0-Final with one domain-controller and two slaves, each having one application_server instance ..

i using web cache container with distributed cache as show below:-

<cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

                    <transport stack="tcp" cluster="${jboss.cluster.group.name}" lock-timeout="300000"/>

                    <distributed-cache name="dist" start="EAGER" batching="true" mode="SYNC" remote-timeout="300000" owners="2" l1-lifespan="0">

                        <locking striping="false" acquire-timeout="60000" concurrency-level="3000"/>

                        <eviction strategy="LIRS" max-entries="1000"/>

                        <expiration max-idle="120000" lifespan="-1"/>

                        <state-transfer timeout="300000"/>

                        <file-store shared="true" preload="true"/>

                    </distributed-cache>

Issue i am facing:-

file location:-  /project/jboss/server/master/domain/servers/DC_NODE1/data/infinispan/web/default-host/context.dat

i have observed the size of "context.dat" file is constently increases which consumes my all disk space and eventully i get OOM error.. and this behaviour i found in slaves also .. i am not sure,  that binary file may be content sessions ..

Please Suggest for the same

Thanks

[johnhpatton]

One thing to note, we also have the following file which corresponds to the <distributed-cache name="dist" setting.  No idea what's creating the context.dat under default-host.

This makes sense:

/project/jboss/server/master/domain/servers/DC_NODE1/data/infinispan/web/dist.dat

This does not:

/project/jboss/server/master/domain/servers/DC_NODE1/data/infinispan/web/default-host/context.dat

[pferraro]

Moving to WildFly forum.

WildFly's web session clustering logic uses both a cache instance per web application and an additional shared cache among all web applications.

This makes sense:

/project/jboss/server/master/domain/servers/DC_NODE1/data/infinispan/web/dist.dat

This cache instance stores a mapping of node address to "instance-id".  This information is used to embed routing information into the session identifier, and is used by load balancers such as mod_cluster, mod_jk, and mod_proxy_balancer.

This does not:

/project/jboss/server/master/domain/servers/DC_NODE1/data/infinispan/web/default-host/context.dat

This cache instance stores the passivated web sessions for a single web application.  The name is constructed using the host name and context path, which, together, are unique for a given web application.

[pferraro]

How many active web sessions are being generated per node?

According to your configuration, Infinispan will begin passivating the least recently used web sessions when more than 500 sessions exist in memory (i.e. max-entries/2, since there are 2 cache entries per session, assuming that you are using the default SESSION granularity).  This number includes web sessions owned by the node *and* remote backups of web sessions from other nodes.  Given owners="2", and assuming even load distribution, each node will only store 250 web sessions of its own in memory before it begins to passivate web sessions.  Unless your web session state is particularly large, it sounds like your max-entries is too low.

I should also note that using <expiration/> in your web session cache, in lieu of <session-timeout/> in web.xml will result in missing HttpSessionListener.sessionDestroyed(...) events, due to [ISPN-694] Create expiration notification for in-memory cache entries - JBoss Issue Tracker and [ISPN-3064] Extend cache expiration notification to cache stores - JBoss Issue Tracker

[pferraro]

The other reason not to use <expiration/> is that infinispan expires cache entries individually.  However, WildFly stores web session across multiple cache entries.  Infinispan expiration does not happen atomically across all entries of a session (since infinispan has no knowledge of this grouping) - thus bad things can potentially happen.  If you want to use Infinispan expiration as a fail-safe measure, that is ok, but make sure it happens well after the configured <session-timeout/>.

[pferraro]

It also occurred to me the the growth of this file is a side effect of the design of the file cache store itself.  This cache store uses a single file, with a memory map of the location per key.  When more space is required, it gets allocated.  If a cache entry is removed from the store, the vacant space is freed, but that just means it is made available internally for other cache entries.  No space is actually freed on disk.  The side effect of this design is that the file size only ever grows or stays the same - this is, it never shrinks.  I'll verify this with someone from the Infinispan team.

[rvansa]

That's true, and it's even a bit worse: there's no defragmentation, therefore, if you store 1000x 999 byte entry, then remove them and try to store 1000 byte entry, you have to allocate again. Space can be reused only by an entry that is of the same size or smaller.

@lokesh: In your configuration, you use file-store shared="true". You must not do that - file store is designed to be a local-only storage accessed by single node.

[ltarley]

Thanks a LOT Paul and Radim for the valuable reply..

@Paul  , active web sessions are being generated per node is 500 (max-active-sessions=500) from jboss-web.xml ...

and i also make a default expiration configuration as mention below..

<distributed-cache name="dist" start="EAGER" batching="true" mode="SYNC" remote-timeout="300000" owners="2" l1-lifespan="0">

                        <locking striping="false" acquire-timeout="60000" concurrency-level="3000"/>

                        <eviction strategy="LIRS" max-entries="300"/>

                        <expiration max-idle="-1" lifespan="-1" interval="-1"/>

                        <state-transfer timeout="300000"/>

                        <file-store shared="true" preload="true"/>

                    </distributed-cache>

@Radim  we'll try the same with file-store shared="flase".

[pferraro]

A few notes:

1. To disable Infinispan-managed expiration, you can simply omit the <expiration/> element.  If you look at our default configuration, we do not specify <expiration/>.  Web session expiration is handled by WildFly, using the configured <session-timeout/> from web.xml..
2. There seems to be some confusion about the meaning of <max-active-sessions/> in jboss-web.xml.  We define active to mean "stored in memory".  If your web application was not <distributable/> a node will begin throwing exceptions if it tries to create more than max-active-sessions at a time.  For <distributable/> web applications, a node can create more web sessions than specified by max-active-sessions, and will passivate the oldest sessions to make room for the new ones. Thus, when I asked how many session are being created on each node, I'm asking specifically how many web sessions your load client creates per node - since the nodes themselves have no self-imposed limit.  So, my question remains, how many web session are you creating per node?
3. <max-active-sessions/> effectively defines a passivation threshold.  It says that a given node will begin evicting web sessions if the number of locally owned web sessions exceeds this value.  Since, by default, each web session corresponds to 2 cache entries, and owners="2" means that half of the cache entries in memory will be owned by some remote node; if you define <max-active-sessions>500</max-active-sessions> then the effective passivation threshold value from infinispan's perspective is equivalent to 2000 cache entries.  Yet, your Infinispan configuration uses <eviction max-entries="300"/>.  This is way too low and is equivalent to <max-active-sessions>75</max-active-sessions>.  In general, I recommend that you use <max-active-sessions/> instead of <eviction/> to control passivation.  If you want to configure Infinispan-managed <eviction/> as a safe-guard, it should be a more conservative value than <max-active-sessions/> (i.e. >2000).

[ltarley]

Thanks Paul for the detailed clarification on the <max-active-session/>  and apologize for the late reply from my end.

here is the mention cli command which provides the session details.

/host=DC_NODE1/server=wildfly-NODE1/deployment=app.war/subsystem=undertow

[domain@DC_HOST:9990 subsystem=undertow] ls

servlet                    context-root=/app  sessions-created=10245

active-sessions=315        server=default-server      virtual-host=default-host

here the session created per node is "session-created=1025"