• • Upgrade in Test lab for Hudson
  • Production Hudson Upgrade
  • Mime-type mappings (move to a separate page along with the backup, copy, mod and del information)
  • Secure login and authenticated user sessions
  • Move hudson private master to another node
  • After upgrade

Upgrade in Test lab for Hudson

hudson_release is a symbolic link to the version of hudson you want to run. So to upgrade hudson you have to:

1. create a dir say hudson_1.124

2. download hudson.war there

3. unzip it in directory "unzipped" (use that name!)

4. copy hudson_release/unzipped/WEB-INF/slave.jar to hudson_release. launchSlave.sh script assumes this jar to be present in hudson_release/slave.jar

5. copy WEB-INF/slave.war to hudson_1.124

6. finally copy jboss-web.xml from older installs to the WEB-INF dir

7. updating the symbolic link will complete upgrade (rm hudson_release; ln -s hudson_124 hudson_release;)

The system of symbolic links will do the rest by redeploying public and internal instances.

Make sure to be aware which files form test_hudson installation are relevant so you don't copy crap to the hudson installation once ready with testing.

Following need to be tested properly:

• E-Mail notification both from public and private hudson instance.

• Publishing for a new project as well as existing project.

• Build of existing projects.

• Build of a project that takes long time which may include either AS testsuite, Portal testsuite etc.

• Junit reporting and its validity.

• Kill stale processes plugin

Production Hudson Upgrade

Process should be updating test_hudson installation and test it works. Then move the files. For the first migration we've had the following lists. Reconsider them on every upgrade!

• Back up files before upgrade

  • @copy list  (try to list all below anyway)

  • bash_profile

  • .bashrc

  • launchSlave.sh

  • launchWinSlave

  • launchWinSlave.sh

  • hudson_workspace - all except workspace subdir to save space

• Copy relevant files from test hudson installation

  • See attached copy.list

• Modify some files for production use

  • See attached mod.list

• Delete unused files after upgrade successful

  • See attached del.list

• Disable all triggers on the public instance

  • Execute "hudson.triggers.Trigger.timer.cancel()" in the system groovy console

For next upgrades we hopefully won't migrate off EAP so probably we'll need to upgrade hudson version and do the modifications described in mod.list after copying from test_hudson.

Mime-type mappings (move to a separate page along with the backup, copy, mod and del information)

To display .log and .jtr files as plain text in the browser the following mime-mapping have to be added to jboss-as/server/default/deploy/jboss-web.deployer/conf/web.xml or on an application level to the hudson war's web.xml

    <mime-mapping>
        <extension>log</extension>
        <mime-type>text/plain</mime-type>
    </mime-mapping>
    <mime-mapping>
        <extension>jtr</extension>
        <mime-type>text/plain</mime-type>
    </mime-mapping>  

We have made changes in jboss-web.deployer/conf/web.xml so that we dont have to merge this change every time we upgrade hudson.

Secure login and authenticated user sessions

To have login information and authenticated user sessions redirected to https, the following needs to be added in hudson/WEB-INF/web.xml whitin <security-constraint> elements (currently only one is present).

    <user-data-constraint>
      <transport-guarantee>CONFIDENTIAL</transport-guarantee>
    </user-data-constraint>

Move hudson private master to another node

1. disable init script on old box and enable on new box after move

2. change host name in start and stop scripts

3. increase FDs available for hudson

   1. in /etc/security/limits.conf

   2. hudson                soft    nofile          40960

   3. hudson                hard    nofile          40960

4. increase memory given to the java process

   1. use large pages and other tune (see http://www.jbossworld.com/downloads/pdf/thursday/JBOSS_10-1050am_JBoss_Enterprise_Application_Platform_Tuning_v2_Andy_Miller.pdf)

   2. configuration for 3G heap and 256M permgen on large pages on 4G box - example below

      1. /etc/security/limits.conf

   hudson                 soft    memlock         3407872
   hudson                 hard    memlock         3407872
   

1. /etc/group - "hugetlb:x:201:hudson"

2. /etc/sysctl.conf

   kernel.shmmax = 4294967296
   vm.nr_hugepages = 1664
   vm.hugetlb_shm_group = 201
   

1. reboot the box so pages can be reserved

2. see profile.d for JAVA options

1. configure hudson master host in ~hudson/.profile.d

   1. export MAX_JAVA_MEM="-Xmx3g -Xms3g -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseLargePages" in additional-box.sh

   2. ant, maven, java, svn, cvs etc. in PATH as for adding a slave

2. Redirect hudson.qa.jboss.com

   1. file a ticket with helpdesk to do that

   2. for the time DNS is being setup and propagating, forwarding from the old master could be setup

      1. echo 1 > /proc/sys/net/ipv4/ip_forward

      2. echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter

      3. iptables -t nat -I PREROUTING -d <old hudson> -p tcp -m tcp --dport 80 -j DNAT --to-destination <new hudson>

      4. iptables -t nat -I POSTROUTING -d <new hudson> -p tcp -m tcp --dport 80 -j SNAT --to-source <old hudson>

      5. iptables -t nat -I OUTPUT -d <old hudson> -p tcp -m tcp --dport 80 -j DNAT --to-destination <new hudson>

3. Maintain sudoers list.

4. Make sure the new host can connect to existing slaves.

5. iptables forward 80 to 8080

   1. iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8585

   2. iptables -t nat -A OUTPUT -o lo  -p tcp --dport 80 -j REDIRECT --to-ports 8585

   3. service iptables save

   4. chkconfig --levels 2345 iptables on

6. if new master is an existing slave, then disable it

   1. during down time edit config.xml and put false in front of the startup command

After upgrade

Update wiki documentation!