event queue size is at 98% threshold

[lovelyliatroim]

Hi Guys,
Im seeing alot of this in my log files,

16:38:40,367 [STREAMING_STATE_TRANSFER sender,10.251.145.205:53173] WARN org.jboss.cache.RegionImpl - putNodeEvent(): eviction node event queue size is at 98% threshold value of capacity: 200000 Region: /realtime/quotes/ You will need to reduce the wakeUpIntervalSeconds parameter.

Now whats happening is this, my app receives a notification to transfer a region to another node, the node transfers the data which can be up to 500,000 records from that node to the other node. After transfer it will look to delete/remove the 500,000 records from its cache. When the deletion occurs I assume it starts creating all the node events on the queue.

Now wakeUpInterval is set to 1 so that cant be made any better/quicker but the queue size is obviously too small, this kind of event should happen rarely, its only if something goes wrong.

What are the implications of increasing the eventQueue size?? Just more memory i would say. What happens when events are fired and cant get on the eviction queue?? Im seeing something that suggests to me that its dropped because after i call the remove for a region, i still see records in that region when it should be empty, does that make sense to you guys?? If not then its something else going on!!

Also is there a difference between cache.removeRegion(FQN) and cache.removeNode(FQN)?? Currently using removeNode(), will try with region and see if it makes any difference, otherwise i will have to up the queueSize.

Thanks for your time,
LL

[wjm]

We're also seeing this error, not out of state transfer, just randomly once in awhile on a busy day. We've noticed that setting eventQueueSize=400000 (for example) doesnt help, we still get the error at 200000. We've tried adjusting down the minTimeToLiveSeconds now, from one day, to one hour.

[lovelyliatroim]

I dont think the problem is related to state transfer, its after i do the state transfer i look to empty the region, this i assume triggers eviction event(s) that get placed on the queue to get evicted.

I then get the above message(s) but what is worrying is that after things have cooled down i till have data items still in the cache for that region. It should be empty which leads me to believe that if the queue is full then the eviction event is discarded.


By the way the removeRegion does nothing for me, region is still there in its entirety. Is there any way of removing a region with many nodes in one go without causing a flood of events that overflows the queue??


Havent had the time to look at this since but its on my todo list.