I too faced problems like this in my near-realtime production applications handling gigabytes of data. And the issues almost solved as we completed the migration to Infinispan. I would strongly recommend to give it a try.
Thanks Girish. Let me try the workaround. Or last option, we are planning to stop and start cache at runtime and then populate the data again.
One more question, did u develop any utility programs to monitor the cache in production? Some kind of query tool. For ex: node count, data specific to business entity etc. We are checking if we there any jboss cache APIs available for this.
Jmx console is not allowed in production for us and in any case it hangs once data size gets bigger.