You should be able to define indexes on reference properties just like any other value index, in which case ModeShape will store the value of the reference as a string in the index (i.e. the node id).
ModeShape doesn't have a nested properties concept (nor does JCR for that matter). Indexes are only used in ModeShape to speed up queries (they are completely optional) so considering that JCR querying doesn't have any special support for reference properties (ModeShape has a REFERENCE constraint extension JCR-SQL2 - ModeShape 5 - Project Documentation Editor) I don't see how having these types of nested indexes would help in any way.
If you want to query across multiple types of nodes which have a certain "relationship", you have to use the equivalent of JCR-joins (JCR-SQL2 - ModeShape 5 - Project Documentation Editor) or use subqueries (which are another ModeShape extension; see JCR-SQL2 - ModeShape 5 - Project Documentation Editor). In the case of REFERENCE properties, subqueries may actually work really well (see the first link for an example)