-
1. Re: consistently increases .dat file and eventually, OOM facing
johnhpatton Jun 10, 2014 10:22 AM (in response to ltarley)One thing to note, we also have the following file which corresponds to the <distributed-cache name="dist" setting. No idea what's creating the context.dat under default-host.
This makes sense:
/project/jboss/server/master/domain/servers/DC_NODE1/data/infinispan/web/dist.dat
This does not:
/project/jboss/server/master/domain/servers/DC_NODE1/data/infinispan/web/default-host/context.dat
-
2. Re: consistently increases .dat file and eventually, OOM facing
pferraro Jun 11, 2014 9:02 AM (in response to johnhpatton)Moving to WildFly forum.
WildFly's web session clustering logic uses both a cache instance per web application and an additional shared cache among all web applications.
This makes sense:
/project/jboss/server/master/domain/servers/DC_NODE1/data/infinispan/web/dist.dat
This cache instance stores a mapping of node address to "instance-id". This information is used to embed routing information into the session identifier, and is used by load balancers such as mod_cluster, mod_jk, and mod_proxy_balancer.
This does not:
/project/jboss/server/master/domain/servers/DC_NODE1/data/infinispan/web/default-host/context.dat
This cache instance stores the passivated web sessions for a single web application. The name is constructed using the host name and context path, which, together, are unique for a given web application.
-
3. Re: consistently increases .dat file and eventually, OOM facing
pferraro Jun 11, 2014 9:22 AM (in response to ltarley)How many active web sessions are being generated per node?
According to your configuration, Infinispan will begin passivating the least recently used web sessions when more than 500 sessions exist in memory (i.e. max-entries/2, since there are 2 cache entries per session, assuming that you are using the default SESSION granularity). This number includes web sessions owned by the node *and* remote backups of web sessions from other nodes. Given owners="2", and assuming even load distribution, each node will only store 250 web sessions of its own in memory before it begins to passivate web sessions. Unless your web session state is particularly large, it sounds like your max-entries is too low.
I should also note that using <expiration/> in your web session cache, in lieu of <session-timeout/> in web.xml will result in missing HttpSessionListener.sessionDestroyed(...) events, due to [ISPN-694] Create expiration notification for in-memory cache entries - JBoss Issue Tracker and [ISPN-3064] Extend cache expiration notification to cache stores - JBoss Issue Tracker
-
4. Re: consistently increases .dat file and eventually, OOM facing
pferraro Jun 11, 2014 10:40 AM (in response to ltarley)The other reason not to use <expiration/> is that infinispan expires cache entries individually. However, WildFly stores web session across multiple cache entries. Infinispan expiration does not happen atomically across all entries of a session (since infinispan has no knowledge of this grouping) - thus bad things can potentially happen. If you want to use Infinispan expiration as a fail-safe measure, that is ok, but make sure it happens well after the configured <session-timeout/>.
-
5. Re: consistently increases .dat file and eventually, OOM facing
pferraro Jun 12, 2014 4:43 PM (in response to johnhpatton)It also occurred to me the the growth of this file is a side effect of the design of the file cache store itself. This cache store uses a single file, with a memory map of the location per key. When more space is required, it gets allocated. If a cache entry is removed from the store, the vacant space is freed, but that just means it is made available internally for other cache entries. No space is actually freed on disk. The side effect of this design is that the file size only ever grows or stays the same - this is, it never shrinks. I'll verify this with someone from the Infinispan team.
-
6. Re: consistently increases .dat file and eventually, OOM facing
rvansa Jun 13, 2014 2:35 AM (in response to pferraro)That's true, and it's even a bit worse: there's no defragmentation, therefore, if you store 1000x 999 byte entry, then remove them and try to store 1000 byte entry, you have to allocate again. Space can be reused only by an entry that is of the same size or smaller.
@lokesh: In your configuration, you use file-store shared="true". You must not do that - file store is designed to be a local-only storage accessed by single node.
-
7. Re: consistently increases .dat file and eventually, OOM facing
ltarley Jun 13, 2014 6:46 AM (in response to rvansa)Thanks a LOT Paul and Radim for the valuable reply..
@Paul , active web sessions are being generated per node is 500 (max-active-sessions=500) from jboss-web.xml ...
and i also make a default expiration configuration as mention below..
<distributed-cache name="dist" start="EAGER" batching="true" mode="SYNC" remote-timeout="300000" owners="2" l1-lifespan="0">
<locking striping="false" acquire-timeout="60000" concurrency-level="3000"/>
<eviction strategy="LIRS" max-entries="300"/>
<expiration max-idle="-1" lifespan="-1" interval="-1"/>
<state-transfer timeout="300000"/>
<file-store shared="true" preload="true"/>
</distributed-cache>
@Radim we'll try the same with file-store shared="flase".
-
8. Re: consistently increases .dat file and eventually, OOM facing
pferraro Jun 13, 2014 8:09 AM (in response to ltarley)A few notes:
- To disable Infinispan-managed expiration, you can simply omit the <expiration/> element. If you look at our default configuration, we do not specify <expiration/>. Web session expiration is handled by WildFly, using the configured <session-timeout/> from web.xml..
- There seems to be some confusion about the meaning of <max-active-sessions/> in jboss-web.xml. We define active to mean "stored in memory". If your web application was not <distributable/> a node will begin throwing exceptions if it tries to create more than max-active-sessions at a time. For <distributable/> web applications, a node can create more web sessions than specified by max-active-sessions, and will passivate the oldest sessions to make room for the new ones. Thus, when I asked how many session are being created on each node, I'm asking specifically how many web sessions your load client creates per node - since the nodes themselves have no self-imposed limit. So, my question remains, how many web session are you creating per node?
- <max-active-sessions/> effectively defines a passivation threshold. It says that a given node will begin evicting web sessions if the number of locally owned web sessions exceeds this value. Since, by default, each web session corresponds to 2 cache entries, and owners="2" means that half of the cache entries in memory will be owned by some remote node; if you define <max-active-sessions>500</max-active-sessions> then the effective passivation threshold value from infinispan's perspective is equivalent to 2000 cache entries. Yet, your Infinispan configuration uses <eviction max-entries="300"/>. This is way too low and is equivalent to <max-active-sessions>75</max-active-sessions>. In general, I recommend that you use <max-active-sessions/> instead of <eviction/> to control passivation. If you want to configure Infinispan-managed <eviction/> as a safe-guard, it should be a more conservative value than <max-active-sessions/> (i.e. >2000).
-
9. Re: consistently increases .dat file and eventually, OOM facing
ltarley Jun 17, 2014 6:31 AM (in response to pferraro)Thanks Paul for the detailed clarification on the <max-active-session/> and apologize for the late reply from my end.
here is the mention cli command which provides the session details.
/host=DC_NODE1/server=wildfly-NODE1/deployment=app.war/subsystem=undertow
[domain@DC_HOST:9990 subsystem=undertow] ls
servlet context-root=/app sessions-created=10245
active-sessions=315 server=default-server virtual-host=default-host
here the session created per node is "session-created=1025"