Jboss cache and heapspace memory

[haribaasha]

I am using a linux machine 4 gb ram, these are my jboss start config:
-Xms:512 -Xmx 1024

i am using jboss cache to cache string objects with a key. these are basically xml string but stored as plain java strings. I have written a customLRUPolicy wherein at the end of day when maxage is reached (86400 seconds) the object is removed from cache rather than evict. otherwise no difference between wat i have patched on LRUPolicy.

I am using bdbje cacheloader. The no of nodes at any time is about 65000. maxnodes is set at 30000 so anything more than tat wud be on persistent storage. The entire cachedirectory(persistent storage) now is about 1.3 G.

Now the java heap space has reached about 1020G and any further writes to cache throws a OutOfMemory error. When i kick in gc there is no reduction in memory at all.

When i disable caching and run my process the usage is constant at 350 M and once over and a gc kicks in it comes down to 80 M.


does this mean all the nodes are in memory ? is there anything i can change in the eviction to solve this or only a increase in heapspace memory will help ?

[mircea.markus]

i am using jboss cache to cache string objects with a key. these are basically xml string but stored as plain java strings. I have written a customLRUPolicy wherein at the end of day when maxage is reached (86400 seconds) the object is removed from cache rather than evict. otherwise no difference between wat i have patched on LRUPolicy.
so insted of calling Cache.evict your CustomLRUPolicy calls Cache.remove, right?
There are two possibilities of doing that, one would be [1]Cache.remove or [2]CacheImpl._remove
For [1], the call goes through the interceptor chain including CacheStore interceptor, which would also remove the node from the persistent storage. This results in not having the data in memory both and disk... Can you please send the used cache configuration file.
[2] is not recommended at all as it violates cache internals (e.g. transaction locks etc)

[haribaasha]

Yes u are right, i call remove instead of evict when maxage is reached.
i am using the Jboss Cache 1.4 , i think jalapeno...so that wud be a remove call on TreeCache. This wud be the (2) u were talking abt ?

my cache config file as in u mean the jboss-service.xml ?

<?xml version="1.0" encoding="UTF-8"?>


jboss:service=Naming



<TCP bind_addr="10.1.30.66"
start_port="51000"
loopback="true"/>

<PING timeout="2000" num_initial_members="3"
up_thread="true" down_thread="true"/>
<TCPPING initial_hosts="10.1.30.62[51000],10.1.30.66[51000]" port_range="3" timeout="3500" num_initial_members="3" up_thread="true" down_thread="true"/>
<MERGE2 min_interval="10000" max_interval="20000"/>
<FD shun="true" up_thread="true" down_thread="true"
timeout="2500" max_tries="5"/>
<VERIFY_SUSPECT timeout="3000" num_msgs="3"
up_thread="true" down_thread="true"/>
<pbcast.NAKACK gc_lag="50"
retransmit_timeout="300,600,1200,2400,4800"
max_xmit_size="8192"
up_thread="true" down_thread="true"/>
<UNICAST timeout="300,600,1200,2400,4800"
window_size="100" min_threshold="10"
down_thread="true"/>
<pbcast.STABLE desired_avg_gossip="20000"
up_thread="true" down_thread="true"/>
<FRAG frag_size="8192"
down_thread="true" up_thread="true"/>
<pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
shun="true" print_local_addr="true"/>
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>





<!--
Node locking level : SERIALIZABLE
REPEATABLE_READ (default)
READ_COMMITTED
READ_UNCOMMITTED
NONE

REPEATABLE_READ
-->
REPL_ASYNC

<!-- Name of cluster. Needs to be the same for all clusters, in order
to find each other
-->
TreeCache

<!-- JGroups protocol stack properties. Can also be a URL,
e.g. file:/home/bela/default.xml

-->
<!--
The max amount of time (in milliseconds) we wait until the
initial state (ie. the contents of the cache) are retrieved from
existing members in a clustered environment
-->
20000

<!-- New 1.3.x cache loader config block -->

<!-- Eviction Policy Configuration -->
org.jboss.cache.eviction.CustomLRUPolicy
<!-- Specific eviction policy configurations. This is LRU -->


5
<!-- Cache wide default -->

300
100
80000


375
125
82000

[haribaasha]

One more thing i noticed. i turned off the caccheloader completely, i.e no bdbjeloader. so when the evict is called it is evicted from in-memory. In this case once my operation is complete and eviction happns the jvm comes back to the state before i started. so here are my results :

1. withcacheloader(bdbje) and CustomLRUPolicy(evicts from cache for timetolive, calls remove on Treecache for maxage) - memory hits very high peak and doesnt come down 1.1G

2. same as 1 only very low time to live, so evict will be called almost instantly , but high max age so the nodes are there in persistent storage - memory usage similar to 1.

3. no cache at all - memory usage is minimum about 100M continously

4. with cache and no cacheloader, hence evicted will be lost - goes up to 800M but on eviction comes down to 100M again.

[haribaasha]

ok, now i had a look at the BDbje code and noticed that it takes about 60 percent of the total memory, which is 60 percent of the 1024 i allocated in Xmx. so it caches all the objects already present in jbosscache and the copy of all this i s also there in the persistent storage.

now je is supposed to take in a je.properties when forming the environment , but looking at bdbjecacheloader we pass the envdir from there as java.io.tmpdir which is /tmp/ directory on unix systems. this is the reason why my putting je.properties everywhere(conf, lib, bin) didnt make any difference to the bdbjeconfig(it kept starting with cachepercent at 60 )

patching the BdbjeCacheloader to set the percentage of memory berkleydb should take for its inmemory cache to a bare minimum solves the problem.
in the method start():
i added this saying bdbje to use only 5 percent of total memory for its inmemory and put the whole thing in persistent.
envConfig.setPercent(5);

now everything works fine.the maxmemory used comes down from 1G to 500M and once the process is over gc kicks in and brings it down to 160M. previously it never even came down from 1G

Hope this helps!!

Hari