Node in eviction queue not seen in printDetails

[jorgemoralespou_2]

Hi,
I have this trace in my logs:

2008-06-26 11:31:41,601 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] Recycle queue is empty
2008-06-26 11:31:41,601 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] process(): region: /_default_/
2008-06-26 11:31:41,601 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] processed 0 node events in region: /_default_/
2008-06-26 11:31:41,601 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] Recycle queue is empty
2008-06-26 11:31:41,601 TRACE [org.jboss.cache.eviction.LRUAlgorithm] Node /THIRD_PARTY_DAILY_COUNTERS/25/32:-:1111 has been idle for 86213470ms
2008-06-26 11:31:41,601 TRACE [org.jboss.cache.eviction.LRUAlgorithm] Node /THIRD_PARTY_DAILY_COUNTERS/25/32:-:1111 should not be evicted
2008-06-26 11:31:41,601 TRACE [org.jboss.cache.eviction.LRUAlgorithm] Node /THIRD_PARTY_DAILY_COUNTERS/25/52:-:1111 has been idle for 86063447ms
2008-06-26 11:31:41,601 TRACE [org.jboss.cache.eviction.LRUAlgorithm] Node /THIRD_PARTY_DAILY_COUNTERS/25/52:-:1111 should not be evicted
2008-06-26 11:31:41,601 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] process(): region: /DELIVERY_STATUS_RETRIES/
2008-06-26 11:31:41,601 TRACE [org.jboss.cache.eviction.BaseEvictionAlgorithm] processed 0 node events in region: /DELIVERY_STATUS_RETRIES/

Here it looks that there is objects in the eviction queue, and when I do a printDetails() from JMX console, I see:

 /DELIVERY_STATUS

 /THIRD_PARTY_MONTHLY_COUNTERS
jboss:internal:uninitialized: null

 /6
jboss:internal:uninitialized: null

 /THIRD_PARTY_DAILY_COUNTERS

I guess that when I have removed those nodes manually (with a schedulerr job) nodes have not been removed from the eviction queue.
I'm using 1.4.1.SP9

Thanks,

[manik]

How does your scheduler job remove stuff? Using a remove() or an evict()?

[jorgemoralespou_2]

Using a remove(Fqn).

As we don`t have a cache loader configured for this cache, we could probably do an evict, but since we have done our own abstraction on top of JBossCache for our way of doing things, we have chosen to use remove so it works with any underlying JBossCache.

CacheEviction is configured the same, so it should evict at same rate, I guess. The thing is that it seems that the is some left over stuff.

Cache config is as follows:

<server>

 <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar" />


 <!-- ==================================================================== -->
 <!-- Defines TreeCache configuration -->
 <!-- ==================================================================== -->


 <mbean code="org.jboss.cache.TreeCache"
 name="jboss.cache:service=SOMServicesDataCache">

 <depends>jboss:service=Naming</depends>
<!--
 <depends>jboss:service=TransactionManager</depends>

 <attribute name="TransactionManagerLookupClass">
 org.jboss.cache.JBossTransactionManagerLookup
 </attribute>
-->

 <!--
 Isolation level : SERIALIZABLE
 REPEATABLE_READ (default)
 READ_COMMITTED
 READ_UNCOMMITTED
 NONE
 -->
 <attribute name="IsolationLevel">NONE</attribute>

 <!--
 Valid modes are LOCAL
 REPL_ASYNC
 REPL_SYNC
 INVALIDATION_ASYNC
 INVALIDATION_SYNC
 -->
<!--
 <attribute name="CacheMode">REPL_SYNC</attribute>
-->
 <attribute name="CacheMode">REPL_ASYNC</attribute>

 <!--
 Just used for async repl: use a replication queue
 -->
 <attribute name="UseReplQueue">false</attribute>

 <!--
 Replication interval for replication queue (in ms)
 -->
 <attribute name="ReplQueueInterval">0</attribute>

 <!--
 Max number of elements which trigger replication
 -->
 <attribute name="ReplQueueMaxElements">0</attribute>

 <!-- Name of cluster. Needs to be the same for all clusters, in order
 to find each other
 -->
 <attribute name="ClusterName">
 SOMServicesCache-Cluster
 </attribute>

 <!-- JGroups protocol stack properties. Can also be a URL,
 e.g. file:/home/bela/default.xml
 <attribute name="ClusterProperties"></attribute>
 -->

 <attribute name="ClusterConfig">
 <config>
 <!-- UDP: if you have a multihomed machine,
 set the bind_addr attribute to the appropriate NIC IP address, e.g bind_addr="192.168.0.2"
 -->
 <!-- UDP: On Windows machines, because of the media sense feature
 being broken with multicast (even after disabling media sense)
 set the loopback attribute to true -->
 <UDP
 mcast_addr="${jboss.cache.SOMServicesCache.addr:228.1.2.3}"
 mcast_port="${jboss.cache.SOMServicesCache.port:48866}" ip_ttl="64"
 ip_mcast="true" mcast_send_buf_size="150000"
 mcast_recv_buf_size="80000" ucast_send_buf_size="150000"
 ucast_recv_buf_size="80000" loopback="false"
 down_thread="false" up_thread="false"/>
 <PING timeout="2000" num_initial_members="3"
 up_thread="false" down_thread="false" />
 <MERGE2 min_interval="10000" max_interval="20000"
 up_thread="false" down_thread="false" />
 <!-- <FD shun="true" up_thread="true" down_thread="true" />-->
 <FD_SOCK down_thread="false" up_thread="false"/>
 <VERIFY_SUSPECT timeout="1500"
 up_thread="false" down_thread="false" />
 <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800"
 max_xmit_size="8192" up_thread="false" down_thread="false"
 discard_delivered_msgs="true"/>
 <UNICAST timeout="600,1200,2400"
 down_thread="false" up_thread="false"/>
 <pbcast.STABLE desired_avg_gossip="20000"
 up_thread="false" down_thread="false" />
 <FRAG frag_size="8192" down_thread="false" up_thread="false" />
 <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
 shun="true" print_local_addr="true"
 down_thread="false" up_thread="false"/>
 <pbcast.STATE_TRANSFER up_thread="true" down_thread="true" />
 </config>
 </attribute>


 <!--
 Whether or not to fetch state on joining a cluster
 NOTE this used to be called FetchStateOnStartup and has been renamed to be more descriptive.
 -->
 <attribute name="FetchInMemoryState">${cache.service-data.fetch-in-memory:true}</attribute>

 <!--
 The max amount of time (in milliseconds) we wait until the
 initial state (ie. the contents of the cache) are retrieved from
 existing members in a clustered environment
 -->
 <attribute name="InitialStateRetrievalTimeout">${cache.service-data.state-retrieval-timeout:15000}</attribute>

 <!--
 Number of milliseconds to wait until all responses for a
 synchronous call have been received.
 -->
 <attribute name="SyncReplTimeout">2000</attribute>

 <!-- Max number of milliseconds to wait for a lock acquisition -->
 <attribute name="LockAcquisitionTimeout">10000</attribute>

 <!-- Name of the eviction policy class. -->
 <attribute name="EvictionPolicyClass">
 org.jboss.cache.eviction.LRUPolicy
 </attribute>

 <!-- Specific eviction policy configurations. This is LRU -->
 <attribute name="EvictionPolicyConfig">
 <config>
 <!-- This attribute will be share by all eviction policies -->
 <attribute name="wakeUpIntervalSeconds">30</attribute>
 <!-- Cache wide default 86400-->
 <region name="/_default_">
 <attribute name="maxNodes">100000</attribute>
 <attribute name="timeToLiveSeconds">86400</attribute>
 </region>
 </config>
 </attribute>


 <!--
 Indicate whether to use region based marshalling or not. Set this to true if you are running under a scoped
 class loader, e.g., inside an application server. Default is "false".
 -->

 <attribute name="UseRegionBasedMarshalling">true</attribute>
 <attribute name="InactiveOnStartup">false</attribute>

 <attribute name="CacheLoaderConfiguration">
 <config>
 <passivation>false</passivation>
 <preload>${cache.service-data.cache-loader.preload:/}</preload>
 <shared>true</shared>

 <cacheloader>
 <class>
 org.jboss.cache.loader.ClusteredCacheLoader
 </class>
 <properties>timeout=${cache.service-data.cache-loader.timeout:30000}</properties>
 <async>true</async>
 <fetchPersistentState>false</fetchPersistentState>
 <ignoreModifications>false</ignoreModifications>

 </cacheloader>

 </config>
 </attribute>


 </mbean>


 <!-- Uncomment to get a graphical view of the TreeCache MBean above -->
 <!-- <mbean code="org.jboss.cache.TreeCacheView" name="jboss.cache:service=TreeCacheView">-->
 <!-- <depends>jboss.cache:service=TreeCache</depends>-->
 <!-- <attribute name="CacheService">jboss.cache:service=TreeCache</attribute>-->
 <!-- </mbean>-->


</server>

We have recently changed from REPL_SYNC to REPL_ASYNC due to problems in our network and retransmissions, so this could be the cause of the difference in numbers, I guess.

We have seen that caches are connected to the JGroups channel and that replication is happening, although we don't have enough log level in our production environment to see if there is some nodes not replicated.

[manik]

Using a remove(Fqn).

Doing a remove should remove events from an eviction queue as well.

[jorgemoralespou_2]

Yeah, as you say, "it hould".

We'll investigate a little deeper. Thanks,

[manik]

:-) If you can prove to me that it doesn't do so - perhaps by way of a more focused and specific test - I'll look into a fix and a release.