hudson_release is a symbolic link to the version of hudson you want to run. So to upgrade hudson you have to:
create a dir say hudson_1.124
download hudson.war there
unzip it in directory "unzipped" (use that name!)
copy hudson_release/unzipped/WEB-INF/slave.jar to hudson_release. launchSlave.sh script assumes this jar to be present in hudson_release/slave.jar
copy WEB-INF/slave.war to hudson_1.124
finally copy jboss-web.xml from older installs to the WEB-INF dir
updating the symbolic link will complete upgrade (rm hudson_release; ln -s hudson_124 hudson_release;)
The system of symbolic links will do the rest by redeploying public and internal instances.
Make sure to be aware which files form test_hudson installation are relevant so you don't copy crap to the hudson installation once ready with testing.
Following need to be tested properly:
E-Mail notification both from public and private hudson instance.
Publishing for a new project as well as existing project.
Build of existing projects.
Build of a project that takes long time which may include either AS testsuite, Portal testsuite etc.
Junit reporting and its validity.
Kill stale processes plugin
Production Hudson Upgrade
Process should be updating test_hudson installation and test it works. Then move the files. For the first migration we've had the following lists. Reconsider them on every upgrade!
Back up files before upgrade
@copy list (try to list all below anyway)
hudson_workspace - all except workspace subdir to save space
Copy relevant files from test hudson installation
See attached copy.list
Modify some files for production use
See attached mod.list
Delete unused files after upgrade successful
See attached del.list
Disable all triggers on the public instance
Execute "hudson.triggers.Trigger.timer.cancel()" in the system groovy console
For next upgrades we hopefully won't migrate off EAP so probably we'll need to upgrade hudson version and do the modifications described in mod.list after copying from test_hudson.
Mime-type mappings (move to a separate page along with the backup, copy, mod and del information)
To display .log and .jtr files as plain text in the browser the following mime-mapping have to be added to jboss-as/server/default/deploy/jboss-web.deployer/conf/web.xml or on an application level to the hudson war's web.xml
<mime-mapping> <extension>log</extension> <mime-type>text/plain</mime-type> </mime-mapping> <mime-mapping> <extension>jtr</extension> <mime-type>text/plain</mime-type> </mime-mapping>
We have made changes in jboss-web.deployer/conf/web.xml so that we dont have to merge this change every time we upgrade hudson.
Secure login and authenticated user sessions
To have login information and authenticated user sessions redirected to https, the following needs to be added in hudson/WEB-INF/web.xml whitin <security-constraint> elements (currently only one is present).
<user-data-constraint> <transport-guarantee>CONFIDENTIAL</transport-guarantee> </user-data-constraint>
Move hudson private master to another node
disable init script on old box and enable on new box after move
change host name in start and stop scripts
increase FDs available for hudson
hudson soft nofile 40960
hudson hard nofile 40960
increase memory given to the java process
use large pages and other tune (see http://www.jbossworld.com/downloads/pdf/thursday/JBOSS_10-1050am_JBoss_Enterprise_Application_Platform_Tuning_v2_Andy_Miller.pdf)
configuration for 3G heap and 256M permgen on large pages on 4G box - example below
hudson soft memlock 3407872 hudson hard memlock 3407872
/etc/group - "hugetlb:x:201:hudson"
kernel.shmmax = 4294967296 vm.nr_hugepages = 1664 vm.hugetlb_shm_group = 201
reboot the box so pages can be reserved
see profile.d for JAVA options
configure hudson master host in ~hudson/.profile.d
export MAX_JAVA_MEM="-Xmx3g -Xms3g -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseLargePages" in additional-box.sh
ant, maven, java, svn, cvs etc. in PATH as for adding a slave
file a ticket with helpdesk to do that
for the time DNS is being setup and propagating, forwarding from the old master could be setup
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter
iptables -t nat -I PREROUTING -d <old hudson> -p tcp -m tcp --dport 80 -j DNAT --to-destination <new hudson>
iptables -t nat -I POSTROUTING -d <new hudson> -p tcp -m tcp --dport 80 -j SNAT --to-source <old hudson>
iptables -t nat -I OUTPUT -d <old hudson> -p tcp -m tcp --dport 80 -j DNAT --to-destination <new hudson>
Maintain sudoers list.
Make sure the new host can connect to existing slaves.
iptables forward 80 to 8080
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8585
iptables -t nat -A OUTPUT -o lo -p tcp --dport 80 -j REDIRECT --to-ports 8585
service iptables save
chkconfig --levels 2345 iptables on
if new master is an existing slave, then disable it
during down time edit config.xml and put false in front of the startup command
Update wiki documentation!