A ModeShape JcrEngine can definitely be embedded within an Eclipse plugin, though I do not know of anyone doing this yet. The ModeShape JARs are OSGi bundles, so IIUC they should be able to be plopped directly into the Eclipse plugin directory and used by your Eclipse plugins.
I also don't know of anyone working on an Eclipse Team provider for ModeShape (or JCR). Offhand, I'm not really sure how well the Eclipse Team mechanism would map to the JCR versioning API. But it's probably worth some thought.
Hope this helps! Best regards,