By "non-volatile", I presume that you mean the repository is configured to persist to the file system, database, or some other storage.
If that's the case, then the repository will definitely NOT load the entire content into memory, ever. Instead, individual nodes are loaded into memory from persistent storage as needed, and they are cached within a workspace cache that uses the LIRS algorithm (an improvement over LRU). Changes to nodes are of course managed within your session (only the delta is stored in memory in your session), and persisted upon save (or upon transaction commit if you're using user transactions). Note that you do have control over the workspace cache configuration.
Binary values are NEVER all loaded into memory, either. In fact, most of the binary storage options will only buffer some of a single binary value only as your application stream through the binary value's content.
There are ways in which ModeShape configured to store everything into memory. The first is a local "volatile" repository that never persists anything; as soon as the repository is shut down, all content is lost. The other is if ModeShape is clustered in distributed mode with no cache store, and in this case all content is actually stored multiple times in memory across the cluster. (This topology is actually not recommended at this time.)