Yes, the connector should implement Pageable, which is the mechanism designed to prevent ModeShape from loading *all* of the children in the parent when the parent node is materialized. We should change our FileSystemConnector to implement Pageable, so feel free to add an enhancement request to our JIRA. We'd welcome any contribution for it, too (if you're interested/able).

The GitConnector is the best example of a Pageable connector, though the concept is pretty straightforward. Just remember that you probably do want to load at least some number of child references when constructing the document for a parent node, although how many is a tradeoff. Loading just a small number (or even 0) child references will make it faster to create the parent node document, but any subsequent access of the child references (e.g., due to client iteration over some chidlren) will require an additional trip to the connector.

Thanks for logging those requests. Making the FileSystemConnector implement Pageable is a great idea, and something we'll definitely do.

However, I'm not convinced that MODE-1983 is the correct change, but we'll definitely take a look. Whenever we have the child's key, we can actually efficiently obtain the child by using `WorkspaceCache.getChildReference(parentKey, nodeKey)'. However, when we only have the name of the child (optionally including the SNS index, whic is assumed to be 1 unless another is specified), the only option is to call `node.getChildReferences(cache).getChild(segment)'.

The bottom line is that I'm not sure that the suggestion will actually help. Anytime we have the key we should be directly looking it up by the key, but if we only have the name we have no choice but to use the second form.

BTW, both methods load the child by materializing the parent node, since the parent is the place where all the child reference (e.g., the key of the child and the name, without SNS index) instances are kept.

• The 'WorkspaceCache.getChildReference(parentKey,nodeKey)' basically gets the parent document and then walks through the ChildReference objects looking for one with a matching key. (This is done to ensure that the node is actually a child of the parent; we used to not actually make this check, but that resulted in problems when one Session was reading children of a node and another thread moved or removed a child.)
• The CachedNode.getChildReferences(cache) merely gets the parent's ChildReferences (plural) container and looks for a ChildReference instance that has a matching name and SNS index.

The representation of the node inside the WorkspaceCache makes these operations more efficient than you might think. First of all, the cached representation (e.g., CachedNode) contains a ChildReferences implementation that, depending upon how the children are stored, will store enough information to quickly find nodes by ID and by name. Most of our implementations are in ImmutableChildReferences, including:

1. the Medium implementation that has a map of children by key and a multi-map keyed by name. (All of the children with the same name are kept in the multi-map under the name; the first has a SNS index of 1, the second 2, the third 3, etc. This is how we don't actually store the SNS index anywhere, since it's implicit in the order of the children.)
2. a Federated implementation of ChildReferences that is used only for the federated nodes (nodes that have both internal and external nodes)
3. a Segment implementation of ChildReferences that is used when a connector is Pageable and when an internal node document stores only some of the children on the parent, with the remainder stored in additional segments (e.g., other documents inside Infinispan cache). This implementation knows how to work efficientlly with the large number of children, stored in different segments.

Right now, all internal nodes are stored with all child references inside the parent's document, and thus Medium implementation is used for all internal nodes. We have code that can run in a background thread that will walk through all persisted documents to find nodes that contain lots of child references and will optimize them by breaking up the child references into multiple segments. When this happens and such a document is materialized, the Segment implementation is used.

However, at this time, we've not enabled the optimization algorithm, primarily because we've not yet had time to tune it to see what segment sizes are ideal. If the segments are too small, ModeShape will need to materialize more smaller documents. If the segments are too big, ModeShape will have to materialize fewer documents, but working with each one will be more expensive.