F17 boxgrinder-meta appliance doesn't seem to be able to grind an F17 boxgrinder-meta appliance

[sweeney]

Hi,

     I finally got my Jenkins boxgrinder-meta appliance to build.  It's a lightly modified version of the stock appliance, with Java and a Jenkins user added, and some minor tweaks to get the privilege escalation working.  The appliance file is as follows:

name: %JOB_NAME%

summary: Jenkins Boxgrinder slave instance

version: %BUILD_NUMBER%

release: 0

hardware:

  memory: 1024

  partitions:

    "/":

      size: 10

appliances:

  - boxgrinder-meta

packages:

  - java-1.7.0-openjdk

files:

  "/":

    - etc/sudoers.d/jenkins

    - home/jenkins/.ssh/authorized_keys

    - tmp/boxgrinder.sed

post:

  base:

    - chown root:root /etc/sudoers.d/jenkins

    - chmod 440 /etc/sudoers.d/jenkins

    - useradd jenkins

    - chown -R jenkins:jenkins /home/jenkins

    - chmod 700 /home/jenkins/.ssh

    - chmod 600 /home/jenkins/.ssh/authorized_keys

    - sed -i -f /tmp/boxgrinder.sed /usr/share/gems/gems/boxgrinder-build-0.10.4/bin/boxgrinder-build

This builds fine on an older F15 boxgrinder-meta version 0.10.2 node in our own datacenter (The build system substitues in values for the things that look like DOS variables).   The freshly ground appliance has box-grinder-build version 0.10.4.  I have started an instance of this appliance in ec2 and successfully attached it to our Jenkins cluster.  In order to validate it, I tried building a vanilla boxgrinder-meta appliance (appliance files from GitHub), and it fails with the following error:

T, [2013-05-02T12:41:04.078796 #13220] TRACE -- : GFS: rootfs / rootfs rw 0 0

proc /proc proc rw,relatime 0 0

/dev/root / ext2 rw,noatime 0 0

/proc

T, [2013-05-02T12:41:04.080200 #13220] TRACE -- : GFS: /proc proc rw,relatime 0 0

/sys /sys sysfs rw,relatime 0 0

/dev /dev devtmpfs rw,relatime,size=141648k,nr_inodes=35412,mode=755 0 0

T, [2013-05-02T12:41:05.751422 #13220] TRACE -- : GFS:   No v

T, [2013-05-02T12:41:05.752761 #13220] TRACE -- : GFS: olume groups found

T, [2013-05-02T12:41:06.828147 #13220] TRACE -- : GFS:  

T, [2013-05-02T12:41:06.830277 #13220] TRACE -- : GFS: No volume groups found

T, [2013-05-02T12:41:07.216894 #13220] TRACE -- : GFS: 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

T, [2013-05-02T12:41:07.219578 #13220] TRACE -- : GFS:     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

T, [2013-05-02T12:41:07.323016 #13220] TRACE -- : GFS:     inet 127.0.0.1/8 brd 127.255.255.255 scope host lo

T, [2013-05-02T12:41:07.325961 #13220] TRACE -- : GFS:     inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

T, [2013-05-02T12:41:07.327833 #13220] TRACE -- : GFS: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

T, [2013-05-02T12:41:07.329286 #13220] TRACE -- : GFS:     link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff

T, [2013-05-02T12:41:07.330816 #13220] TRACE -- : GFS:     inet 169.254.2.10/16 brd 169.254.255.255 scope global eth0

T, [2013-05-02T12:41:07.332763 #13220] TRACE -- : GFS:     inet6 fe80::5054:ff:fe12:3456/64 scope link

       valid_lft forever preferred_lft forever

T, [2013-05-02T12:41:07.334156 #13220] TRACE -- : GFS:

T, [2013-05-02T12:41:07.458946 #13220] TRACE -- : GFS: default via 169.254.2.2 dev eth0

T, [2013-05-02T12:41:07.470498 #13220] TRACE -- : GFS: 169.254.0.0/16 dev eth0  proto kernel  scope link  src 169.254.2.10

T, [2013-05-02T12:41:07.751941 #13220] TRACE -- : GFS: Module                  Size  Used by

T, [2013-05-02T12:41:07.770012 #13220] TRACE -- : GFS: kvm_amd                59705  0

T, [2013-05-02T12:41:07.771712 #13220] TRACE -- : GFS:

T, [2013-05-02T12:41:07.780738 #13220] TRACE -- : GFS: kvm                   438094  1 kvm_amd

T, [2013-05-02T12:41:07.784938 #13220] TRACE -- : GFS: i2c_piix4              22106  0

T, [2013-05-02T12:41:07.788867 #13220] TRACE -- : GFS: i2c_core               38353  1 i2c_piix4

T, [2013-05-02T12:41:07.793365 #13220] TRACE -- : GFS: virtio_net             27892  0

T, [2013-05-02T12:41:07.795713 #13220] TRACE -- : GFS: virtio_scsi            18006  0

T, [2013-05-02T12:41:07.798723 #13220] TRACE -- : GFS: virtio_blk             18325  1

T, [2013-05-02T12:41:07.801536 #13220] TRACE -- : GFS: virtio_rng             13117  0

T, [2013-05-02T12:41:07.865136 #13220] TRACE -- : GFS: virtio_balloon         13508  0

T, [2013-05-02T12:41:07.869780 #13220] TRACE -- : GFS: virtio_mmio            13157  0

T, [2013-05-02T12:41:07.872878 #13220] TRACE -- : GFS: sparse_keymap          13526  0

T, [2013-05-02T12:41:07.876056 #13220] TRACE -- : GFS: rfkill                 21736  0

T, [2013-05-02T12:41:07.878787 #13220] TRACE -- : GFS: sym53c8xx              76601  0

T, [2013-05-02T12:41:07.882185 #13220] TRACE -- : GFS: scsi_transport_spi     30237  1 sym53c8xx

T, [2013-05-02T12:41:07.885186 #13220] TRACE -- : GFS: crc8                   12750  0

T, [2013-05-02T12:41:07.888184 #13220] TRACE -- : GFS: crc_ccitt              12613  0

T, [2013-05-02T12:41:07.890839 #13220] TRACE -- : GFS: crc_itu_t              12613  0

T, [2013-05-02T12:41:07.893473 #13220] TRACE -- : GFS: libcrc32c              12603  0

T, [2013-05-02T12:41:08.109801 #13220] TRACE -- : GFS: Thu May  2 12:41:00 EDT 2013

T, [2013-05-02T12:41:08.121044 #13220] TRACE -- : GFS: uptime:

T, [2013-05-02T12:41:08.283117 #13220] TRACE -- : GFS: 16.09 0.88

T, [2013-05-02T12:41:08.964743 #13220] TRACE -- : GFS: verbose daemon enabled

T, [2013-05-02T12:41:08.969275 #13220] TRACE -- : GFS: linux commmand line: panic=1 console=ttyS0 udevtimeout=300 no_timer_check acpi=off printk.time=1 cgroup_disable=memory selinux=1 enforcing=0 guestfs_verbose=1 TERM=unknown n

T, [2013-05-02T12:41:08.970603 #13220] TRACE -- : GFS: oapic

T, [2013-05-02T12:41:09.733026 #13220] TRACE -- : GFS: udevadm settle

T, [2013-05-02T12:41:10.001896 #13220] TRACE -- : GFS: udevd[54]: sender uid=65534, message ignored

T, [2013-05-02T12:41:10.003555 #13220] TRACE -- : GFS:

T, [2013-05-02T12:41:10.030779 #13220] TRACE -- : GFS: udevsettle

T, [2013-05-02T12:41:10.049304 #13220] TRACE -- : GFS: udevsettle: No such file or directory

T, [2013-05-02T12:41:10.118908 #13220] TRACE -- : GFS: [45538ms] appliance is up

D, [2013-05-02T12:41:10.119152 #13220] DEBUG -- : GFS: launch = 0

D, [2013-05-02T12:41:10.119276 #13220] DEBUG -- : GFS: list_devices

T, [2013-05-02T12:41:10.124098 #13220] TRACE -- : GFS: guestfsd: main_loop: new request, len 0x28

D, [2013-05-02T12:41:10.133314 #13220] DEBUG -- : GFS: list_devices = ["/dev/vda"]

D, [2013-05-02T12:41:10.133579 #13220] DEBUG -- : GFS: list_partitions

T, [2013-05-02T12:41:10.136355 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 7 (list_devices) took 0.01 seconds

guestfsd: main_loop: new request, len 0x28

D, [2013-05-02T12:41:10.209503 #13220] DEBUG -- : GFS: list_partitions = ["/dev/vda1", "/dev/vda2"]

T, [2013-05-02T12:41:10.209746 #13220] TRACE -- : Mounting partitions...

D, [2013-05-02T12:41:10.209900 #13220] DEBUG -- : GFS: list_partitions

T, [2013-05-02T12:41:10.212103 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 8 (list_partitions) took 0.01 seconds

guestfsd: main_loop: new request, len 0x28

D, [2013-05-02T12:41:10.214408 #13220] DEBUG -- : GFS: list_partitions = ["/dev/vda1", "/dev/vda2"]

D, [2013-05-02T12:41:10.219612 #13220] DEBUG -- : GFS: vfs_type "/dev/vda1"

T, [2013-05-02T12:41:10.221009 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 8 (list_partitions) took 0.00 seconds

T, [2013-05-02T12:41:10.223053 #13220] TRACE -- : GFS: guestfsd: main_loop: new request, len 0x38

T, [2013-05-02T12:41:10.225675 #13220] TRACE -- : GFS: blkid -c /dev/null -o value -s TYPE /dev/vda1

D, [2013-05-02T12:41:10.473473 #13220] DEBUG -- : GFS: vfs_type = ""

D, [2013-05-02T12:41:10.473791 #13220] DEBUG -- : GFS: vfs_type "/dev/vda2"

T, [2013-05-02T12:41:10.476996 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 198 (vfs_type) took 0.13 seconds

guestfsd: main_loop: new request, len 0x38

T, [2013-05-02T12:41:10.479242 #13220] TRACE -- : GFS: blkid -c /dev/null -o value -s TYPE /dev/vda2

D, [2013-05-02T12:41:10.926133 #13220] DEBUG -- : GFS: vfs_type = "ext4"

D, [2013-05-02T12:41:10.926375 #13220] DEBUG -- : GFS: vfs_type "/dev/vda2"

T, [2013-05-02T12:41:10.929368 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 198 (vfs_type) took 0.17 seconds

guest

T, [2013-05-02T12:41:10.930598 #13220] TRACE -- : GFS: fsd: main_loop: new request, len 0x38

T, [2013-05-02T12:41:10.932344 #13220] TRACE -- : GFS: blkid -c /dev/null -o value -s TYPE /dev/vda2

D, [2013-05-02T12:41:11.292403 #13220] DEBUG -- : GFS: vfs_type = "ext4"

T, [2013-05-02T12:41:11.292815 #13220] TRACE -- : Mounting /dev/vda2 partition to /...

D, [2013-05-02T12:41:11.292929 #13220] DEBUG -- : GFS: mount_options "" "/dev/vda2" "/"

T, [2013-05-02T12:41:11.295354 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 198 (vfs_type) took 0.18 seconds

guestfsd: main_

T, [2013-05-02T12:41:11.296581 #13220] TRACE -- : GFS: loop: new request, len 0x44

T, [2013-05-02T12:41:11.305086 #13220] TRACE -- : GFS: mount -o  /dev/vda2 /sysroot/

T, [2013-05-02T12:41:11.932311 #13220] TRACE -- : GFS: [   17.732460] EXT4-fs (vda2): mounted filesystem with ordered data mode. Opts: (null)

D, [2013-05-02T12:41:12.055808 #13220] DEBUG -- : GFS: mount_options = 0

D, [2013-05-02T12:41:12.061952 #13220] DEBUG -- : GFS: set_e2label "/dev/vda2" "79d3d2d4"

T, [2013-05-02T12:41:12.063288 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 74 (mount_options) took 0.33 seconds

T, [2013-05-02T12:41:12.126216 #13220] TRACE -- : GFS: guestfsd: main_loop: new request, len 0x44

T, [2013-05-02T12:41:12.128929 #13220] TRACE -- : GFS: blkid -c /dev/null -o value -s TYPE /dev/vda2

T, [2013-05-02T12:41:12.519844 #13220] TRACE -- : GFS: e2label /dev/vda2 79d3d2d4

D, [2013-05-02T12:41:12.956901 #13220] DEBUG -- : GFS: set_e2label = 0

T, [2013-05-02T12:41:12.957102 #13220] TRACE -- : Partition mounted.

D, [2013-05-02T12:41:12.957235 #13220] DEBUG -- : GFS: exists "/etc/sysconfig/selinux"

T, [2013-05-02T12:41:12.961100 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 80 (set_e2label) took 0.44 seconds

T, [2013-05-02T12:41:13.026641 #13220] TRACE -- : GFS:

guestfsd: main_loop: new request, len 0x44

D, [2013-05-02T12:41:13.349594 #13220] DEBUG -- : GFS: exists = 1

T, [2013-05-02T12:41:13.349809 #13220] TRACE -- : Loading SElinux policy...

D, [2013-05-02T12:41:13.349959 #13220] DEBUG -- : GFS: aug_init "/" 32

T, [2013-05-02T12:41:13.352055 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 36 (exists) took 0.15 seconds

guestfsd: main_loop: new request, len 0x34

D, [2013-05-02T12:41:34.871279 #13220] DEBUG -- : GFS: aug_init = 0

D, [2013-05-02T12:41:34.871488 #13220] DEBUG -- : GFS: aug_rm "/augeas/load//incl[. != '/etc/sysconfig/selinux']"

T, [2013-05-02T12:41:34.873402 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 16 (aug_init) took 9.16 seconds

guestfsd: main_loop: new request, len 0x60

D, [2013-05-02T12:41:34.893188 #13220] DEBUG -- : GFS: aug_rm = 294

D, [2013-05-02T12:41:34.893399 #13220] DEBUG -- : GFS: aug_load

T, [2013-05-02T12:41:34.895290 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 22 (aug_rm) took 0.02 seconds

guestfsd: main_loop: new request, len 0x28

D, [2013-05-02T12:41:35.616497 #13220] DEBUG -- : GFS: aug_load = 0

D, [2013-05-02T12:41:35.616718 #13220] DEBUG -- : GFS: aug_get "/files/etc/sysconfig/selinux/SELINUX"

T, [2013-05-02T12:41:35.708450 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 27 (aug_load) took 0.30 seconds

guestfsd: main_loop:

T, [2013-05-02T12:41:35.710549 #13220] TRACE -- : GFS:  new request, len 0x50

T, [2013-05-02T12:41:35.712785 #13220] TRACE -- : GFS: guestfsd: error: no matching node

D, [2013-05-02T12:41:35.712928 #13220] DEBUG -- : GFS: aug_get = NULL (error)

F, [2013-05-02T12:41:35.713358 #13220] FATAL -- : Guestfs::Error: aug_get: no matching node

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:219:in `aug_get'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:219:in `load_selinux_policy'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:202:in `execute'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:174:in `block in customize'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:123:in `block (2 levels) in initialize_guestfs'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:169:in `prepare_guestfs'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:123:in `block in initialize_guestfs'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:77:in `log_callback'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:123:in `initialize_guestfs'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:173:in `customize'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/image-helper.rb:129:in `customize'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/plugins/os/rpm-based/rpm-based-os-plugin.rb:76:in `build_with_appliance_creator'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/plugins/os/fedora/fedora-plugin.rb:53:in `execute'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/plugins/base-plugin.rb:172:in `run'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:184:in `execute_plugin'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:200:in `block in execute_with_userchange'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/util/permissions/user-switcher.rb:27:in `call'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/util/permissions/user-switcher.rb:27:in `change_user'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:199:in `execute_with_userchange'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:127:in `block in execute_plugin_chain'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:125:in `each'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:125:in `execute_plugin_chain'

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:164:in `create'

/usr/share/gems/gems/boxgrinder-build-0.10.4/bin/boxgrinder-build:203:in `<top (required)>'

/usr/bin/boxgrinder-build:23:in `load'

/usr/bin/boxgrinder-build:23:in `<main>'

I searched the web for similar errors and all I could find was this from the libguestfs mailing list last January:

https://www.redhat.com/archives/libguestfs/2013-January/msg00079.html

The libguestfs people seem to think that this is a boxgrinder bug.

The file /etc/selinux/config file on my appliance exists and has SELINUX=disabled

I'm happy to attach any further files to help diagnose the problem.

[sweeney]

I mounted the partially built appliance disk image, and the /etc/sysconfig/selinux file in the target file system there is as follows:

[root@localhost ~]# cd /mnt/boxgrinder-meta

[root@localhost boxgrinder-meta]# cat etc/sysconfig/selinux

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

#       enforcing - SELinux security policy is enforced.

#       permissive - SELinux prints warnings instead of enforcing.

#       disabled - SELinux is fully disabled.

SELINUX=permissive

# SELINUXTYPE= type of policy in use. Possible values are:

#       targeted - Only targeted network daemons are protected.

#       strict - Full SELinux protection.

SELINUXTYPE=targeted

[root@localhost boxgrinder-meta]#

[sweeney]

So, I loaded the augeas command line tool on both the F15 boxgrinder-meta in our data center and on the F17 node in AWS, and the problem seems to be that /etc/sysconfig/selinux is no longer in the Shellvars lens in the F17 version.

F15:

[root@boxgrinder2 ~]# augtool

augtool> print /augeas/load//incl[. = '/etc/sysconfig/selinux']

/augeas/load/Shellvars/incl[15] = "/etc/sysconfig/selinux"

augtool>

F17:

augtool> print /augeas/load//incl[. = '/etc/sysconfig/selinux']

augtool>

So, F15 can grind an F17 appliance, but F17 can't.

[sweeney]

It turns out that on both F15 and F17, the file /etc/sysconfig/selinux is in fact a symbolic link:

[root@boxgrinder2 ~]# ls -li /etc/selinux/config /etc/sysconfig/selinux

400501 -rw-r--r--. 1 root root 449 Feb 17  2012 /etc/selinux/config

400502 lrwxrwxrwx. 1 root root  17 Feb 17  2012 /etc/sysconfig/selinux -> ../selinux/config

[root@boxgrinder2 ~]#

This appears to also be true on CentOS5 and CentOS6 and presumably other headgear distros

The Augeas Shellvars lens in F15 includes /etc/sysconfig/selinux, whereas the F17 version has only /etc/selinux/config

If I change the three occurrences of etc/sysconfig/selinux to etc/selinux/config in /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb, then the (hacked) F17 boxgrinder-meta appliance can correctly build an F17 boxgrinder-meta appliance.  This hack would, however, fail if run on F15, with the same problem I was experiencing earlier. 

/usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb