SWR API Changes: Resolution Strategies, Transitivity, and Filters

[alrubinger]

The last meeting on ShrinkWrap Resolvers was a discussion on the grammars for resolving with regards to transitivity and post-resolution filtering.  While we've each been reviewing the current API to best understand the problem domain, I think it's important that we frame these discussions based upon a clean slate and our ideal solution.  In other words, identifying the relevant use cases and proposing psudocode to fit them. 

I'd also like to value the following concepts, in order:

* Support for All Features in Current API

* Clear Behaviour

* Concision

...meaning that we should expose all current features currently in place, we should make it clear to the user, and trim verbosity as much as possible without reducing the readability.

The following are some questions raised during the discussion.

1) Make resolution a pluggable mechanism?

In this scenario we make the assumption that resolution is more than simply "transitive or not transitive", and that we cannot predict all user requirements.  Note that this might bleed partially into the notion of filtering, a post-resolution step which permits exclusion based on some criteria.  However it may be beneficial for us to merge the concepts of merging and resolution strategy.  So in that case we'd supply a bunch of canned "ResolutionStrategy" implementations, which are an encapsulation of the rules for inclusion.

Something like:

artifact.resolveAs(ArchiveType.class).using(ResolutionStrategy argument);

2) Require the user to specify a ResolutionStrategy?

I like the grammar above because it becomes clear to the user exactly what strategy is in place; no question will be raised as to whether transitive deps will be sucked in, for example.  This necessarily makes things more verbose, but I think the case is warranted, because I don't see how we can pick an intelligent default.

3) Kill the Shortcut API

I like shortcuts, but not separate entry points.  2 entry points make it difficult to switch between one style of supplying the call and another, and make for a more difficult-to-learn API.  So the only shortcut I really like is a shorthand for selecting the MavenDependencyResolver.

OK, let's see some use cases.

a) Resolve a single artifact by GAV

b) Resolve a single artifact and all of its runtime dependencies by GAV

c) Resolve a single artifact by GA, and version from a POM

d) Resolve all dependencies declared in a POM

e) Resolve all dependencies declared in a POM, with some conditional exclusions

WDYT?

S,

ALR

[kpiwko]

I like 1) and 2). As for 3), it does save user a lot of typing, so if we want to kill it, we need to make resolution of a single artifact easier. E.g. a Strategy which returns a single artifact instead of Array or Collection.

I don't like to word all in resolve all dependencies. It is not clear to me what it means, e.g. is it all as for compile phase (compile, provided), or rather a test phase (complile, provided, test) , or even package (compile, runtime) phase? All these

are valid use cases to me.

I've placed another list of the use case here: https://gist.github.com/2502823

[alrubinger]

I like 1) and 2). As for 3), it does save user a lot of typing, so if we want to kill it, we need to make resolution of a single artifact easier.

+1

E.g. a Strategy which returns a single artifact instead of Array or Collection.

I was actually thinking something along the lines of JPA?

http://docs.oracle.com/javaee/6/api/javax/persistence/Query.html#getSingleResult()

I don't like to word all in resolve all dependencies. It is not clear to me what it means, e.g. is it all as for compile phase (compile, provided), or rather a test phase (complile, provided, test) , or even package (compile, runtime) phase?

+1

I've placed another list of the use case here: https://gist.github.com/2502823

Cool.  I wasn't clear with the use cases though; I presented them so we could write what we thought it should look like (as you've nicely done in the Gist).

S,

ALR

[dan.j.allen]

Andrew Rubinger wrote:

3) Kill the Shortcut API

I like shortcuts, but not separate entry points.  2 entry points make it difficult to switch between one style of supplying the call and another, and make for a more difficult-to-learn API.  So the only shortcut I really like is a shorthand for selecting the MavenDependencyResolver.

I'd tread carefully here. One of the driving forces for adoption of alterante JVM languages (and even languages outside the JVM) is shorthand. I agree that we shouldn't have behavior divergence, but I think the shorthand is going to make or break ShrinkWrap Resolver acceptance.

In short, you need both. You need the direct route (you know, all those worn out paths through the grass on a college campus...pave them) and you also need an API that is extensible (and subsequently wrapped in other shorthands downstream).

I think you can anticipate the ~80% case. It's grabbing a single artifact with or without transitives. After that, there is a long tail of special requirements that the formal API can masterfully accomodate.

[dan.j.allen]

Andrew Rubinger wrote:

E.g. a Strategy which returns a single artifact instead of Array or Collection.

I was actually thinking something along the lines of JPA?

http://docs.oracle.com/javaee/6/api/javax/persistence/Query.html#getSingleResult()

+1 This is one of the most popular methods in all of JPA. (Don't feel locked in to taking exactly the same path, but a method name that reflects the use case is important). iterator().next() is a sore thumb inside business logic.

Personally, I prefer the first() method from Ruby's Enumerable. I think it's the most accurate to what you are really doing, which is taking the first one, expecting one, but ignoring extras if they are present.

[alrubinger]

Dan Allen wrote:

I'd tread carefully here. One of the driving forces for adoption of alterante JVM languages (and even languages outside the JVM) is shorthand. I agree that we shouldn't have behavior divergence, but I think the shorthand is going to make or break ShrinkWrap Resolver acceptance.

I do agree.  Here I just meant "Kill the Shortcut API" as a wholly separate API (which in its current state is not compatible with the full API).  They have separate entry points.  In the IRC session today we discussed adding shorthand notations for common cases, which in turn delegate to the longhand way.  But still: one entry point.

S,

ALR

[alrubinger]

Dan Allen wrote:

Personally, I prefer the first() method from Ruby's Enumerable. I think it's the most accurate to what you are really doing, which is taking the first one, expecting one, but ignoring extras if they are present.

Hmm, I think I prefer "getSingleResult"-style approaches.  Because here the user is really expecting only one thing to be resolved, and it's appropriate to throw an exception if that's not the case and he/she has mucked up the query.  "first", by contrast, infers that they just want to get the first result from a set.

S,

ALR

[dan.j.allen]

Andrew Rubinger wrote:

Dan Allen wrote:

Personally, I prefer the first() method from Ruby's Enumerable. I think it's the most accurate to what you are really doing, which is taking the first one, expecting one, but ignoring extras if they are present.

Hmm, I think I prefer "getSingleResult"-style approaches.  Because here the user is really expecting only one thing to be resolved, and it's appropriate to throw an exception if that's not the case and he/she has mucked up the query.  "first", by contrast, infers that they just want to get the first result from a set.

Got it. In that case, go for it.

[dan.j.allen]

Andrew Rubinger wrote:

Dan Allen wrote:

I'd tread carefully here. One of the driving forces for adoption of alterante JVM languages (and even languages outside the JVM) is shorthand. I agree that we shouldn't have behavior divergence, but I think the shorthand is going to make or break ShrinkWrap Resolver acceptance.

I do agree.  Here I just meant "Kill the Shortcut API" as a wholly separate API (which in its current state is not compatible with the full API).  They have separate entry points.  In the IRC session today we discussed adding shorthand notations for common cases, which in turn delegate to the longhand way.  But still: one entry point.

Excellent news.

[alrubinger]

I've put up a POC for the new API:

  https://github.com/shrinkwrap/resolver/commit/df787e173f2e33ed76ec5df26237e5d5b17bbf46

We'll align further work on this area in upstream/SHRINKRES-39, and I'll keep this rebased off master.  For the time being, I'll handle all pushes to this upstream branch (so pull requests if ya got work you wanna throw in there).

I've taken the Gist that Karel prepared (which outlines our use cases) and have begun to fulfill these on the new API.  That test case is called UseCasesTestCase, and is located:

  https://github.com/shrinkwrap/resolver/blob/SHRINKRES-39/impl-maven-archive-prototype/src/test/java/org/jboss/shrinkwrap/resolver/impl/maven/prototyping/UseCasesTestCase.java

Some noteable changes:

1) As Karel and I discovered in the last IRC meeting, resolution is really a series of steps and it has a logical flow.  In plain English, I found that I'd describe the process like:

  "I want to (optionally configure with these settings and) resolve these coordinates using this resolution strategy and get the result in this format"

That grammar seemed to fit all of our use cases.  So we have what I'm currently calling "Stages" in the API.  The first stage allows the user to supply the desired coordinate(s), the second to choose the resolution strategy (ie. transitivity, scopes, exclusions, etc), and the third to choose the resultant format (File, InputStream, Archive, something pluggable).

As the test case will show, the result looks like:

    @Test
    public void singleArtifactAsFile() {
        @SuppressWarnings("unused")
        final File longhand = DependencyResolvers.use(MavenResolverSystem.class).resolve("G:A:V").using(NonTransitiveResolutionStrategy.class).asSingle(File.class);

        @SuppressWarnings("unused")
        final File shortcut = Maven.RESOLVER.resolve("G:A:V").withoutTransitivity().asSingle(File.class);
    }

The reason a user would opt for the longhand over the short is to supply pluggable ResolutionStrategies or ResolverSystems, etc.  And the shorthand is simply API delegation to known impls we supply which are commonly-used.

2) I've realized that SWR doesn't really have any relationship at all with ShrinkWrap archives until we hit the "formatting" stage.  So I decoupled out any SW dependencies from SWR.  I think they should be brought together again in an integration layer which adds the capability to resolve "as(JavaArchive.class)".  This way SWR becomes a useful standalone API wrapper above Aether, again, with no ties to SW (or any other compile dependencies in the API for that matter)

3) You'll note that while I have documented the API that's there, it's only as complete as the UseCasesTestCase shows.  I've marked all areas that are still under development as //TODO in this case.  Specifically, I still need to:

• Account for exclusions, scopes (and other stuff that used to be done in Filters)
• Figure out how the pluggable ResolutionStrategy mechanism will work
• Put in place the features currently served by EffectivePomMavenResolver (like "importAllDependencies")

4) No changes have been made to the existing SWR modules; all work is currently in new modules named "*-prototype".

Looking forward to some feedback and keeping the momentum going.

S,

ALR

[silenius]

@SuppressWarnings("unused")
final File shortcut = Maven.RESOLVER.resolve("G:A:V").withoutTransitivity().asSingle(File.class);

I believe that asSingle() already assumes no transitive dependencies.

Is it not possible to describe the process like this?

"I want to (optionally configure with these settings and), resolve these coordinates, get the result in this format and use this resolution strategy (if not defined assume a default)"

That way, the strategy will work more like a filter and we can make it optional allowing us to do something like this:

@SuppressWarnings("unused")
final File shortcut = Maven.RESOLVER.resolve("G:A:V").asSingle(File.class);

The method as() can also assume a default strategy in the same way.

What do you think?

[alrubinger]

Samuel Santos wrote:

I believe that asSingle() already assumes no transitive dependencies.

I'd explored that, but hit the use case where we want to resolve two dependencies without transitivity:

final File[] shorthand = Maven.RESOLVER.resolve("G:A:V", "G2:A2:V2").withoutTransitivity().as(File.class);

To date I've found no way to infer a single result or not based on ResolutionStrategy, which necessitates "as" and "asSingle".  Any other ideas welcome!

S,

ALR

[silenius]

But you can always do as follows:

final File[] shorthand = Maven.RESOLVER.resolve("G:A:V", "G2:A2:V2").as(File.class).withoutTransitivity();

and still support the option:

final File shortcut = Maven.RESOLVER.resolve("G:A:V").asSingle(File.class);

[silenius]

I take back what I said.

The only option would be to do as we currently to with filters, which is not so fluent as you'd like:

final File[] shorthand = Maven.RESOLVER.resolve("G:A:V", "G2:A2:V2").as(File.class, MyStrategy.class);

[alrubinger]

I see you're looking to make the StrategyStage optional.  But how does that work in the API?  Your example above has "asSingle(File)" returning "File"...now how would that also allow us to chain in a call to ".withoutTransitivity()"?

S,

ALR