We have a proposed enhancement (in the SingleFileStore) that optimizes space usage and attempts to reclaim it whenever possible.
It is two fold:
1. Truncates the file if there are free fileentries at the end of the data file.
2. Coalesces consecutive free fileentries into one so that the probability of finding a free file entry capable enough to contain a newly added key-value pair increases.
Attached is the modified extended version of the source. We would like to get it reviewed from the relevant authors.
We have tested this internally and it seems to be helping with space usage.
SingleFileStore.java 25.9 KB
The file store should compact space, unless the size of the entries you add is constantly increasing. Do you produce a test that reproduces this behaviour?
Yes, we have two usecases:
1. A persistent queue where entries get created and removed very fast. The size of data in this case is random.
In our environment, the sizes of such files grow to around ~700 MB before the file stops growing.
With our proposed fix (ISPN-3877), the size stays very much within control ~5-6 MB
2. A result repository for a certain tasks. As each task progresses, it updates the result in the repository (against its id as key).
Each time, the result size grows. So, new results always get appended towards end of the file (and earlier entry in the cache gets added to freelist).
These smaller entries are then useless for allocating the larger objects. Coalescing smaller free entries into bigger ones makes this possible.
ISPN-3894 is created to track this.
I have created two issues and two pull requests:
corresponding pull request: https://github.com/infinispan/infinispan/pull/2335
corresponding pull request: https://github.com/infinispan/infinispan/pull/2336
Thanks for taking it so far, looking into this..
Thanks Mircea, the two issues also have tests to reproduce.