Eviction queue fills up causing threads to block
jeremystone Jul 8, 2009 10:23 AMWe have a problem whereby we have seen the following message logged continuously on a customer's installation of our software that uses JBoss Cache with Hibernate:
Warning putNodeEvent(): eviction node event queue size is at 98% threshold value of capacity: 200000 Region: /query/entity/by_name You will need to reduce the wakeUpIntervalSeconds parameter.
We have seen this only once so far. It does not appear to recur after a server restart.
A stack dump shows many threads waiting to put nodes on to the eviction queue. The following is representative...
Thread: pool-21-thread-4 : priority:5, demon:false, threadId:144, threadState:WAITING, lockName:java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@584b2d sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254) org.jboss.cache.RegionImpl.registerEvictionEvent(RegionImpl.java:249) org.jboss.cache.RegionImpl.registerEvictionEvent(RegionImpl.java:234) org.jboss.cache.interceptors.EvictionInterceptor.registerEvictionEventToRegionManager(EvictionInterceptor.java:252) ...etc org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118) org.hibernate.impl.SessionImpl.list(SessionImpl.java:1596) org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306) org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:328) ...etc
The EvictionTimer thread appears just to waiting for something to do...
Thread: EvictionTimer-0 : priority:5, demon:true, threadId:88, threadState:WAITING, lockName:java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@353fbf sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) java.util.concurrent.DelayQueue.take(DelayQueue.java:160) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:619)
The only inkling as to what is occurring is that perhaps an eviction algorithm (silently) threw an unchecked exception causing the EvictionTimerTask Task to no longer be scheduled.
(see http://java.sun.com/javase/6/docs/api/java/util/concurrent/ScheduledExecutorService.html#scheduleWithFixedDelay(java.lang.Runnable,%20long,%20long,%20java.util.concurrent.TimeUnit)
which states '...If any execution of the task encounters an exception, subsequent executions are suppressed....').
Even if this is not the cause of our problem then EvictionTimerTask seems susceptible to this behaviour. Maybe you should consider using something like the Spring Framework DelegatingExceptionProofRunnable
(http://static.springsource.org/spring/docs/2.5.6/api/org/springframework/scheduling/support/DelegatingExceptionProofRunnable.html).
Any thoughts on this?