1 Reply Latest reply on May 10, 2010 9:44 AM by pgier

    Our Feedbacks about Hudson, Maven and deployment

    freddy33 May 10, 2010 8:37 AM

       

      Hi,

       

      Since we (JFrog) worked on Hudson, Maven, and Repository Manager environment for many years,
      I think we have some interesting feedbacks about your build and Nexus issues.
      I wanted also to expose the differences between Nexus and Artifactory for the record on this public Mailing list.

       

      Here are the issues I'm talking about:
      1. Mod dav or http(s) deployment URL
      2. Managing the distributionManagement XML tag in pom.xml
      3. Using password for Maven deployment
      4. Timeout and performance issues
      5. Maven metadata XML issue
      6. General feedbacks

       

      1) Mod dav or http(s) deployment URL
      To solve the timeout on deploy issue for richfaces, the recommendation was to use pure https instead of dav:https as url deployment.
      Here are all the caveeat you need to know about Maven deploy wagons:
      • There are 3 main kind of Wagon used with Maven: Lightweight http (the default today), Heavyweight http (the default for older Maven versions), Dav.
      • Dav uses transactions when deploying, it has a better management of memory on the client side, and has a bug calculating checksums. Dav is a must when deploying big files. Now, the issue with dav is you can get timeout due to transactions, so the recommendation is correct!
      • Lightweight http(s) included in Maven, cache the whole byte array in memory so cannot work for big deployment.
      • Dav and Heavyweight protocols suffer from a default configuration that transfer each deployed file twice over the net. This is due to standard HTTP protocol always trying to send a request without authentication, then with it. Check this to find the way to configure preeemptive http authentication in Maven simple http client.
      • Maven is caching credentials information based on URL. It means that if you are authenticating the read queries, the same read credentials are used for deploy! See "Configuring Authentication" here.


      Basically, looking at the above options nothing is satisfactory with standard Maven.

      To solve this issue, we (JFrog) created a build client plugin that manages deployment of the artifacts using http client with preemptive authentication and optimized response time. For info: This client is used to deployed full DVD images of 4.5Gb to Artifactory multiple times per day, from multiple Hudson servers and agents...

       

      2) Managing the distributionManagement XML tag in pom.xml
      To manage correctly all the above parameters, and the deployed URL of the exact repository location for each projects, you need to configure the distributionManagement tag in the pom.xml. This generates a lot of friction, and makes rebuilds of SVN tags almost impossible.
      For good reason you want to split now the different locations were projects should deploye their artifacts. So, all the previous builds and SVN tags are obsolete and cannot be reproduced...
      We believe that managing deployment configuration parameters should be done outside Maven, and that's why we have an Artifactory-Hudson plugin that does exactly this. Your pom.xml are clean of repository manager URLs and you can tuned deployment parameters (Timeout, http or https, ...) in Hudson outside Maven.
      BTW: The Artifactory-Hudson plugin provides a lot more features like full bidirectional traceability between Artifactory and Hudson for all the artifacts produced and used during each hudson job execution.

       

      3) Using password for Maven deployment
      Like you describe here having encrypted password in your settings.xml is a little annoying but it works. Now, you need to know that it still forces you to use https since the clear text password will pass on the net. Now, the issue with Repository Manager password is that all kind of automation scripts beside maven needs them (Hudson, auto-backup, rsync, ...), and so you always find yourself in a tricky situation, see this and that.
      To solve this issue, we implemented an encrypted password system that works as follow:
      • Artifactory generates an asymmetric key pair for each user,
      • Entering your clear text password in the UI gives you a block of the settings.xml with your personal encrypted password,
      • The above encrypted password can be used anywhere (not only maven), and allows only REST API access (maven deploy and so on) not UI access,
      • With the encrypted password, https to do maven deploy is not a must anymore => a lot less load on the servers,
      • Stealing the encrypted password or the private key (kept in Artifactory DB) has minimal security impact compare to full clear text decryption of all the passwords used on the JBoss Hudson server,
      • You can enforce to always use encrypted password for REST and maven deploy.

       

      4) Timeout and performance issues

      This issue is related to the dav mode like described before, to JVM memory parameters, OS open file handles tuning, but also to the next issue about maven-metadata.xml files.
      From our experience, one issue with latest 1.6 Sun JVM is the miscalculation of young memory size. So, when updating/upgrading your server you should verify that -XX:NewSize and -XX:MaxNewSize are at least set to 2/3 of the total heap size. Artifactory (like Nexus) needs a lot of memory space per request (managing files), and so the young eden space of the JVM is the most important parameter. Just be careful to set a good multiple of mega bytes has the JVM has a small bug moving memory blocks from young to old gen space.

      From what I read here, this is what I'll recommend: "-server -Xms1g -Xmx1g -XX:PermSize=128m -XX:MaxPermSize=128m -XX:NewSize=512m -XX:MaxNewSize=512m"

       

      5) Maven metadata XML issue

      This issue is a little tricky to explain, I hope I'll be clear

      For each maven pom or jar file of a SNAPSHOT version, maven needs to 2 maven-metadata.xml files. I'll take the richfaces issue as an example:
      Deploying: org/richfaces/ui/modal-panel/3.3.4-SNAPSHOT/modal-panel-3.3.4-SNAPSHOT.pom needs to update:
      - org/richfaces/ui/modal-panel/3.3.4-SNAPSHOT/maven-metadata.xml to update the timestamp and or the latest unique version number of this version
      - org/richfaces/ui/modal-panel/maven-metadata.xml to update the list of available SNAPSHOT versions and the latest one created

      The issue is that the maven client is doing this complicated work, combined with the fact that maven-metadata.xml files are very repeatedly requested by maven to build its version resolution scheme.

      So in your case, for every module of richfaces 3.3.X-SNAPSHOT, maven deploy the jar (twice due to issue 1), the pom, the sources, read the previous values of maven-metadata.xml on both folders, then write them down (twice due to issue 1). Of course due to dav and the heavy read on these files, you get timeouts.
      The performance tuning you are planning here will help solve this issue since the reads will be offloaded to nginx.
      Be aware, you will hit a bigger problem due to this above design: Repeated corruption of maven-metadata.xml files!

      Like everyone, most of your projects have upstream and downstream dependencies in Hudson. For example: richfaces-jsf2 depends on richfaces. So, it is common that multiple separate SNAPSHOT hudson job (let say richfaces-3.3.4-SNAPSHOT and richfaces-3.3.5-SNAPSHOT) will get executed in parallel. As you can see from the above maven design, the calculation of maven-metadata.xml files will almost certainly fail.

       

      To solve this issue, our deployment system (in the hudson plugin) knows that Artifactory is a smart repository manager and so sends only the pom, jars and verified checksum to the repository manager. Artifactory is then queuing maven metadata recalculation tasks asynchronously, and so avoiding parallelism conflicts and timeout. Implementing this mechanism was far from evident, but we have today integration tests deploying 5 different snapshot versions concurrently (and repeatedly) without a hitch

       

       

      6) General feedbacks
      As you saw, on failed Hudson jobs trying to deploy to Nexus, you end up with half deployed projects. It means that until the successful build 572 of richfaces, the repository contained a random list of good and bad jars and pom for the same project/version. It is recommended to actually do the deployment only until the full build is successful.
      Deployment needs to be a post build action of Hudson (like it is done in our plugin).

       

      Since maven is calculating and deploying the checksum to your Nexus instance, you have general issues like:
      • Playing with dav may have generated a lot of bad checksums that you don't know of;
      • There is no bad checksum verification on deploy, and so the build will be successful but users will get a wrong checksum error;
      • Since the correspondance between hudson and maven checksums could not be trusted you'll lost traceability between Hudson and Nexus.
      For info: Artifactory is capable of failing the deployment if checksums does not match, and has many checksums policy configuration.

      I hope you'll solve your build infrastructure issues for the good of JBoss developers and users