5 Replies Latest reply on Sep 29, 2009 4:10 AM by jeremystone

    Eviction queue fills up causing threads to block

    jeremystone

      We have a problem whereby we have seen the following message logged continuously on a customer's installation of our software that uses JBoss Cache with Hibernate:

      Warning putNodeEvent(): eviction node event queue size is at 98% threshold value of capacity: 200000 Region: /query/entity/by_name You will need to reduce the wakeUpIntervalSeconds parameter.


      We have seen this only once so far. It does not appear to recur after a server restart.

      A stack dump shows many threads waiting to put nodes on to the eviction queue. The following is representative...

      Thread: pool-21-thread-4 : priority:5, demon:false, threadId:144, threadState:WAITING, lockName:java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@584b2d
      
      sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
      java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
      java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
      org.jboss.cache.RegionImpl.registerEvictionEvent(RegionImpl.java:249)
      org.jboss.cache.RegionImpl.registerEvictionEvent(RegionImpl.java:234)
      org.jboss.cache.interceptors.EvictionInterceptor.registerEvictionEventToRegionManager(EvictionInterceptor.java:252)
      ...etc
      org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118)
      org.hibernate.impl.SessionImpl.list(SessionImpl.java:1596)
      org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306)
      org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:328)
      ...etc
      

      The EvictionTimer thread appears just to waiting for something to do...

      Thread: EvictionTimer-0 : priority:5, demon:true, threadId:88, threadState:WAITING, lockName:java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@353fbf
      
      sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
      java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
      java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
      java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
      java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
      java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      java.lang.Thread.run(Thread.java:619)


      The only inkling as to what is occurring is that perhaps an eviction algorithm (silently) threw an unchecked exception causing the EvictionTimerTask Task to no longer be scheduled.
      (see http://java.sun.com/javase/6/docs/api/java/util/concurrent/ScheduledExecutorService.html#scheduleWithFixedDelay(java.lang.Runnable,%20long,%20long,%20java.util.concurrent.TimeUnit)
      which states '...If any execution of the task encounters an exception, subsequent executions are suppressed....').

      Even if this is not the cause of our problem then EvictionTimerTask seems susceptible to this behaviour. Maybe you should consider using something like the Spring Framework DelegatingExceptionProofRunnable
      (http://static.springsource.org/spring/docs/2.5.6/api/org/springframework/scheduling/support/DelegatingExceptionProofRunnable.html).

      Any thoughts on this?