Regarding Question 1 above:

Since we use Virtual IPs we configured a custom $/etc/jetty.xml which specifies a host that is the VIP.

We've temporarily eliminated this custom configuration and now we receive the error printed at the bottom of this post.

Looking in file:/home/our-VIP/.m2/repository/the/groupId/the-artifactId/1.0-SNAPSHOT we see just one file called resolver-status.properties.

This contains:

maven-metadata-repos1.xml.error=

maven-metadata-repos0.xml.error=

maven-metadata-repos1.xml/default-http\://fabric-host\:8181/maven/download/.lastUpdated=1357295801640

maven-metadata-repos1.xml.lastUpdated=1357298894987

maven-metadata-repos0.xml.lastUpdated=1357295302142

maven-metadata-ocado-maven-snapshots.xml.lastUpdated=1357295302548

maven-metadata-repos2.xml.error=

maven-metadata-repos2.xml.lastUpdated=1357295321954

maven-metadata-ocado-maven-snapshots.xml.error=

load/)\: Error transferring file\: Connection refused

maven-metadata-ocado-maven-snapshots.xml.error=

maven-metadata-repos2.xml.error=

Any suggestions what's going wrong? As mentioned in the OP these errors we see relate to a dependency of the FAB we specify, therefore we know that is resolved successfully and retrieved from Nexus. The Connection refused mentioned in the above file refers to the http://fabric-host:8181/maven/download URI so some configuration is still amiss and causing this URI to be inaccessible.

In case it's relevant, our test container setup is simply a lone ensemble node root which has the fmc profile applied to it. The container we attempt to provision with our FAB is a child container of root.

Here's the error we see in FMC, mentioned at the top of this post:

Provision Exception Trace: java.io.IOException: Could not find artifact the.groupId:the-artifactId:jar:1.0-SNAPSHOT in repos0 (file:/home/our-VIP/.m2/repository/)

    at org.fusesource.fabric.fab.osgi.internal.FabResolverFactoryImpl$FabResolverImpl.getInfo(FabResolverFactoryImpl.java:260)

    at org.fusesource.fabric.agent.DeploymentAgent.updateDeployment(DeploymentAgent.java:517)

    at org.fusesource.fabric.agent.DeploymentAgent.doUpdate(DeploymentAgent.java:428)

    at org.fusesource.fabric.agent.DeploymentAgent$1.run(DeploymentAgent.java:238)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

    at java.util.concurrent.FutureTask.run(FutureTask.java:138)

    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

    at java.lang.Thread.run(Thread.java:662)

Caused by: org.sonatype.aether.resolution.ArtifactResolutionException: Could not find artifact the.groupId:the-artifactId:jar:1.0-SNAPSHOT in repos0 (file:/home/our-VIP/.m2/repository/)

    at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:488)

    at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:190)

    at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:250)

    at org.fusesource.fabric.fab.MavenResolverImpl.resolveFile(MavenResolverImpl.java:187)

    at org.fusesource.fabric.fab.DependencyTree.newInstance(DependencyTree.java:139)

    at org.fusesource.fabric.fab.DependencyTree.newInstance(DependencyTree.java:131)

    at org.fusesource.fabric.fab.DependencyTreeResult.getTree(DependencyTreeResult.java:49)

    at org.fusesource.fabric.fab.MavenResolverImpl.collectDependencies(MavenResolverImpl.java:383)

    at org.fusesource.fabric.fab.MavenResolverImpl.collectDependenciesFromPom(MavenResolverImpl.java:291)

    at org.fusesource.fabric.fab.MavenResolverImpl.collectDependencies(MavenResolverImpl.java:263)

    at org.fusesource.fabric.fab.osgi.internal.FabResolverFactoryImpl$FabResolverImpl.collectDependencyTree(FabResolverFactoryImpl.java:149)

    at org.fusesource.fabric.fab.osgi.internal.FabClassPathResolver.resolve(FabClassPathResolver.java:136)

    at org.fusesource.fabric.fab.osgi.internal.FabResolverFactoryImpl$FabResolverImpl.configureInstructions(FabResolverFactoryImpl.java:283)

    at org.fusesource.fabric.fab.osgi.internal.FabResolverFactoryImpl$FabResolverImpl.createInstructions(FabResolverFactoryImpl.java:275)

    at org.fusesource.fabric.fab.osgi.internal.FabResolverFactoryImpl$FabResolverImpl.getInfo(FabResolverFactoryImpl.java:245)

    ... 9 more

Caused by: org.sonatype.aether.transfer.ArtifactNotFoundException: Could not find artifact the.groupId:the-artifactId:jar:1.0-SNAPSHOT in repos0 (file:/home/our-VIP/.m2/repository/)

    at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:854)

    at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:849)

    at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$GetTask.flush(WagonRepositoryConnector.java:602)

    at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$GetTask.flush(WagonRepositoryConnector.java:596)

    at org.sonatype.aether.connector.wagon.WagonRepositoryConnector.get(WagonRepositoryConnector.java:355)

    at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:407)

    ... 23 more

Thank you davsclaus - we're aware of this documentation and have followed it. We know that the maven proxy setup is correct (at the very least mostly correct) because as mentioned the original FAB artifact we specify must be resolved correctly from our Maven proxy, since this error we see relates to a dependency of it that can only be known by fabric from the FAB's POM.

EDIT: We searched the page you linked for maven/download and found nothing so dismissed it - however upon further inspection it mentions an analagous maven/upload URI so we'll now read it more carefully. Thank you. I don't recall this information being on the page when I've visited it previously, I wonder whether we missed it or whether it's been updated.

Another update: the VIP's home directory .m2/repository location contains a directory tree for the FAB we specify, but not the JAR or POM. Instead, this artifact's folder also contains a resolver-status.properties equivalent to that we see for its dependency.

It contains some other files as well:

    maven-metadata-repos1.xml

    maven-metadata-repos1.xml.sha1

    maven-metadata-repos2.xml

    maven-metadata-repos2.xml.sha1

These contain metadata which could only have been identified from Nexus, namely the snapshot versions. For example, the second XML file listed above contains:

<?xml version="1.0" encoding="UTF-8"?>

<metadata modelVersion="1.1.0">

  <groupId>com.anon</groupId>

  <artifactId>the-artifact</artifactId>

  <version>1.0-SNAPSHOT</version>

  <versioning>

    <snapshot>

      <timestamp>20121231.142650</timestamp>

      <buildNumber>1</buildNumber>

    </snapshot>

    <lastUpdated>20121231142650</lastUpdated>

    <snapshotVersions>

      <snapshotVersion>

        <extension>jar</extension>

        <value>1.0-20121231.142650-1</value>

        <updated>20121231142650</updated>

      </snapshotVersion>

      <snapshotVersion>

        <extension>pom</extension>

        <value>1.0-20121231.142650-1</value>

        <updated>20121231142650</updated>

      </snapshotVersion>

    </snapshotVersions>

  </versioning>

</metadata>

We'd really appreciate help with this, since as mentioned we can't find any documentation on the maven/download URI nor how it's configured. As well as searching the doc PDFs I mentioned in a previous post I've run a find . | xargs -n 1 grep 'maven/download' in the Fabric installation directory and nothing was found besides messages in the logs.

EDIT: Well some Googling has finally  turned up the /maven/download URI. We'll now inspect this class to glean what we can.

EDIT2: Not much that helps us in there unfortunately. We can see what the class is doing but it sheds no light on our problem.

Thanks for replying, Stan.

We tried changing the resolver on root to the VIP's IP address through FMC and re-provisioning the FAB container to no avail. We also tried this after deleting the artifact in the VIP's ~/.m2/repository local repo but again it didn't work.

As a failsafe we're going to try recreating the fabric from scratch (it's a breeze with our Puppet + Cucumber setup) and set the resolver as params to fabric:create

Quick question about this:

There are two params available on fabric:create - one is --global-resolver and the other is --resolver.

We want to specify the manualip for each of these.

Does this require a --manual-ip argument for both parameters? The help documentation for fabric:create is unclear.

So I'm asking if this is correct:

--global-resolver manualip --manual-ip 1.2.3.4 --resolver manualip 1.2.3.4

UPDATE:

Well we've tried the above and the parameter construction is correct it seems.

The resolver appears correctly in FMC as manualip and the Manual IP entry is what we've specified, which is certainly the VIP's current IP address.

Sadly we receive the same error as before, indicating the FAB's dependency can't be found in the VIP's ~/.m2/repository. Once again this local repo contains a directory tree for both the FAB artifact and this dependency, yet each only contains a resolver-status.properties under it along with the XML metadata I mentioned earlier for the FAB:

Provision Exception Trace: java.io.IOException: Could not find artifact com.anon:FAB-dependency-artifactId:jar:1.0-SNAPSHOT in repos0 (file:/home/theVIP/.m2/repository/)

    at org.fusesource.fabric.fab.osgi.internal.FabResolverFactoryImpl$FabResolverImpl.getInf

<snip>...</snip>

Caused by: org.sonatype.aether.resolution.ArtifactResolutionException: Could not find artifact com.anon:FAB-dependency-artifactId:jar:1.0-SNAPSHOT in repos0 (file:/home/theVIP/.m2/repository/)

Ok so we may be making progress.

When we redeployed the fabric with the VIP's IP manually specified as the resolver, we hadn't deleted the local artifact directory trees under the VIP's ~/.m2/repository.

We tried this before recreating the fabric and it made no difference.

However, with this fresh fabric deployment with the manual IP specified (which initially made no difference as mentioned in my previous post) we've just deleted the maven local repo dir trees for the FAB artifact and its dependency, and now we see the following.

It looks like we've reached the next stage and this is stating that it can't find appropriate versions of these dependencies, correct?

These listed packages aren't dependencies of our FAB, but may be transitive dependencies; I'll have a look now.

Provision Exception Trace: java.lang.Exception: Can not resolve feature:

Unsatisfied requirement(s):

-

package:(&(package=javax.transaction.xa)(version>=5.5.1)(!(version>=6.0.0)))

com.anon.our-fab

package:(&(package=org.apache.activemq)(version>=5.5.1)(!(version>=6.0.0)))

com.anon.our-fab

package:(&(package=javax.jms)(version>=5.5.1)(!(version>=6.0.0)))

com.anon.our-fab

    at org.fusesource.fabric.agent.ObrResolver.resolve(ObrResolver.java:200)

    at org.fusesource.fabric.agent.DeploymentAgent.updateDeployment(DeploymentAgent.java:554)

    at org.fusesource.fabric.agent.DeploymentAgent.doUpdate(DeploymentAgent.java:428)

    at org.fusesource.fabric.agent.DeploymentAgent$1.run(DeploymentAgent.java:238)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

    at java.util.concurrent.FutureTask.run(FutureTask.java:138)

    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

    at java.lang.Thread.run(Thread.java:662)

Great, thank you for confirming this Stan, and for giving us the missing link! What would we do without you... cry in a corner probably.

Is it the case that each OSGi-fied dependency of the FAB is translated to a feature by Fabric?

We'll consider what you've kindly informed us regarding resolvers below, and respond once we've digested this.

Thank you for this explanation, Stan. We've now removed the global resolver parameter and left only the local one.

We'll try bumping the log level now and see what information it provides. Presumably as you say in your other post, all we need is the missing feature repos now. We'll try to find these and add them to the default profile, this seems more sensible than adding other profiles since there's no overhead from additional features running. Please let me know if you think otherwise.

The FAB is an existing application JAR we have running in Production, so is known to deploy successfully, thank you for the suggestion though.

EDIT: Ah, given your comment above "..maps to ESB features... I'll try with the ESB profile as you recommended instead

Ok so there's no difference with the fuse-esb-minimal profile added, however with fuse-esb-medium added (which has one additional features repo in it compared with ..-minimal) one of the dependency errors disappears. This is good news, it's evidence we're close to getting the FAB installed! There are no additional feature repos in fuse-esb-full compared to ..-medium hence why we didn't try it.

We'll now seek public feature/bundle repositories containing the dependencies we need, or else manually install them as you've said.

Do you know of a list of such feature/bundle repositories? I've read somewhere that bundle repos exist for many common Java dependencies, but a quick Google search for 'karaf feature repositories' hasn't turned up anything useful.

We'd like to reserve the manual installation route as a last resort, since we'll be creating unnecessary work for ourselves in the future if we're missing many dependency features which are available in a public repository we're unaware of.

Presumably all those you know about have been added to the fuse-esb profiles.

EDIT: We seem to be finding a few OBRs from Googling at least. One of the best is on the Apache Felix site.

EDIT2: Ah, it seems a "virtual OBR" is the way forward. See  here.