Custom connectors framework 2.x and 3.x: versioning and search question

[vasilievip]

After looking at forums and jira I'm bit confused with Modeshape custom connectors framework:

• https://issues.jboss.org/browse/MODE-1200
• https://community.jboss.org/thread/194692
• https://issues.jboss.org/browse/MODE-1188
• https://community.jboss.org/message/714193#714193

Here is my understanding of what is there:

- Custom connectors framework in 2.x supports serach only if external repository is fully indexed

- Custom connectors framework in 2.x does not support native search using search features of external repository

- Custom connectors framework in 2.x does not support versioning from external repository but this is targeted probably for 3.1 for brand new implementation

I'm not concerned regarding versioning (but if someone set expectations this would be just great) but I'm confused with native search in external systems w/o building full index of external repository.

How complex would be to hack something for 2.x to support native search (at least some basic function)? If this feature will be available in 3.x?

Thanks in advance,

Ivan

[rhauch]

Here is my understanding of what is there:

- Custom connectors framework in 2.x supports serach only if external repository is fully indexed

- Custom connectors framework in 2.x does not support native search using search features of external repository

These two are accurate statements. The indexes are maintained within ModeShape, and ModeShape 2.x does not push down queries to the connectors. We always had the intension of doing this, but doing so was more complicated than we'd time to work on it. It's fairly obvious how submitting a query to a single connector might work, but querying a workspace that is backed by several connectors (e.g., federated case) makes it very difficult to merge the results from all of the connectors; we'd have to assume the scores from different connectors are comparable (which is probably not a valid assumption). Plus, it doesn't help that most of the connectors don't actually have the ability to query or search.

- Custom connectors framework in 2.x does not support versioning from external repository but this is targeted probably for 3.1 for brand new implementation

This statement is not completely accurate. Any content from any connector can be versioned as long as the nodes are "mix:versionable", but the versions are maintained within ModeShape's version history. So from a client's perspective, the content is versionable. But the connectors actually are no involved whatsoever in this versioning, which is a huge disadvantage for connector that talk to systems that do their own versioning. (Even if we'd added versioning support to the connector SPI, each connector that supported it's own versioning would have to reconcile differences in the semantics of JCR versioning vs the external system's versioning.)

... but I'm confused with native search in external systems w/o building full index of external repository.

Because ModeShape maintains its own indexes, it must index the external content before it can be searched. A connector can generate an event that would kick in the indexing system, but none of them do. (Part of the problem is that it'd have to use NODE_ADDED events, which aren't really accurate with respect to other listeners.) The workaround with 2.x then is to simply invoke the "reindex" methods on the "org.modeshape.jcr.api.Workspace" interface (a subtype of the "javax.jcr.Workspace" interface returned from the "Session.getWorkspace()" method).

How complex would be to hack something for 2.x to support native search (at least some basic function)? If this feature will be available in 3.x?

I think it would be very difficult (if not impossible) to do, and there's no guarantee that even if a connector supported queries that ModeShape would actually push down the query to the connector, since that code path has never been tested or used.

If this feature will be available in 3.x?

To be honest, I think ModeShape 3.1 (and after) will have to support connectors that can and cannot query their own content. We obviously need to fix the auto-indexing problem for connectors that do not support executing queries against their own content. But we do also want to allow connectors to implement their own search/query functionality, and we'll have to assume/require that the scores of the connector's search results were comparable to the scores from ModeShape-supplied results. (Again, this is important when the query is executed across content where some of the content is owned by external systems and the rest by ModeShape.)

I don't know if 3.1 will include a connector SPI with query and versioning support, or whether those features will come in subsequent releases. But the goal is that we do have those features.

Hope that helps

[vasilievip]

Thank you Randall, that helped

I think I'll try to dance around auto-indexing+federation: like add more sources pointing to the sub-trees in external repository and some buttons like - re-index and enable auto-indexing.

[vasilievip]

Randall,

Since MODE-1200 is resolved as won't fix, should I submit new feature request for 3.1 with request to support native search for custom connectors? (So you will have separate item to track)

Thanks

[rhauch]

Since MODE-1200 is resolved as won't fix, should I submit new feature request for 3.1 with request to support native search for custom connectors?

Yes, please.