SingleFileStore contains no rocket science, it's just a ConcurrentHashMap of keys -> entry value offsets. When preloading data, the file is sequentially read and this map is built naively.
When you have cluster with multiple nodes and one of them goes down, you shouldn't do any preloading - the data should be really moved by state transfer. If you use local storage (as single file store, or LevelDB store), you should reboot the node with purge=true to remove old entries.
I'd guess that it's the state transfer that takes a while - you can tune it with the chunkSize attribute and see if increasing that reduces the ST duration.