3 Replies Latest reply on May 10, 2013 8:42 AM by sweeney

    F17 boxgrinder-meta appliance doesn't seem to be able to grind an F17 boxgrinder-meta appliance

    sweeney

      Hi,

       

           I finally got my Jenkins boxgrinder-meta appliance to build.  It's a lightly modified version of the stock appliance, with Java and a Jenkins user added, and some minor tweaks to get the privilege escalation working.  The appliance file is as follows:

       

      name: %JOB_NAME%

      summary: Jenkins Boxgrinder slave instance

      version: %BUILD_NUMBER%

      release: 0

      hardware:

        memory: 1024

        partitions:

          "/":

            size: 10

      appliances:

        - boxgrinder-meta

      packages:

        - java-1.7.0-openjdk

      files:

        "/":

          - etc/sudoers.d/jenkins

          - home/jenkins/.ssh/authorized_keys

          - tmp/boxgrinder.sed

      post:

        base:

          - chown root:root /etc/sudoers.d/jenkins

          - chmod 440 /etc/sudoers.d/jenkins

          - useradd jenkins

          - chown -R jenkins:jenkins /home/jenkins

          - chmod 700 /home/jenkins/.ssh

          - chmod 600 /home/jenkins/.ssh/authorized_keys

          - sed -i -f /tmp/boxgrinder.sed /usr/share/gems/gems/boxgrinder-build-0.10.4/bin/boxgrinder-build

       

      This builds fine on an older F15 boxgrinder-meta version 0.10.2 node in our own datacenter (The build system substitues in values for the things that look like DOS variables).   The freshly ground appliance has box-grinder-build version 0.10.4.  I have started an instance of this appliance in ec2 and successfully attached it to our Jenkins cluster.  In order to validate it, I tried building a vanilla boxgrinder-meta appliance (appliance files from GitHub), and it fails with the following error:

       

      T, [2013-05-02T12:41:04.078796 #13220] TRACE -- : GFS: rootfs / rootfs rw 0 0

      proc /proc proc rw,relatime 0 0

      /dev/root / ext2 rw,noatime 0 0

      /proc

      T, [2013-05-02T12:41:04.080200 #13220] TRACE -- : GFS: /proc proc rw,relatime 0 0

      /sys /sys sysfs rw,relatime 0 0

      /dev /dev devtmpfs rw,relatime,size=141648k,nr_inodes=35412,mode=755 0 0

      T, [2013-05-02T12:41:05.751422 #13220] TRACE -- : GFS:   No v

      T, [2013-05-02T12:41:05.752761 #13220] TRACE -- : GFS: olume groups found

      T, [2013-05-02T12:41:06.828147 #13220] TRACE -- : GFS:  

      T, [2013-05-02T12:41:06.830277 #13220] TRACE -- : GFS: No volume groups found

      T, [2013-05-02T12:41:07.216894 #13220] TRACE -- : GFS: 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

      T, [2013-05-02T12:41:07.219578 #13220] TRACE -- : GFS:     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

      T, [2013-05-02T12:41:07.323016 #13220] TRACE -- : GFS:     inet 127.0.0.1/8 brd 127.255.255.255 scope host lo

      T, [2013-05-02T12:41:07.325961 #13220] TRACE -- : GFS:     inet6 ::1/128 scope host

             valid_lft forever preferred_lft forever

      T, [2013-05-02T12:41:07.327833 #13220] TRACE -- : GFS: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

      T, [2013-05-02T12:41:07.329286 #13220] TRACE -- : GFS:     link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff

      T, [2013-05-02T12:41:07.330816 #13220] TRACE -- : GFS:     inet 169.254.2.10/16 brd 169.254.255.255 scope global eth0

      T, [2013-05-02T12:41:07.332763 #13220] TRACE -- : GFS:     inet6 fe80::5054:ff:fe12:3456/64 scope link

             valid_lft forever preferred_lft forever

      T, [2013-05-02T12:41:07.334156 #13220] TRACE -- : GFS:

      T, [2013-05-02T12:41:07.458946 #13220] TRACE -- : GFS: default via 169.254.2.2 dev eth0

      T, [2013-05-02T12:41:07.470498 #13220] TRACE -- : GFS: 169.254.0.0/16 dev eth0  proto kernel  scope link  src 169.254.2.10

      T, [2013-05-02T12:41:07.751941 #13220] TRACE -- : GFS: Module                  Size  Used by

      T, [2013-05-02T12:41:07.770012 #13220] TRACE -- : GFS: kvm_amd                59705  0

      T, [2013-05-02T12:41:07.771712 #13220] TRACE -- : GFS:

      T, [2013-05-02T12:41:07.780738 #13220] TRACE -- : GFS: kvm                   438094  1 kvm_amd

      T, [2013-05-02T12:41:07.784938 #13220] TRACE -- : GFS: i2c_piix4              22106  0

      T, [2013-05-02T12:41:07.788867 #13220] TRACE -- : GFS: i2c_core               38353  1 i2c_piix4

      T, [2013-05-02T12:41:07.793365 #13220] TRACE -- : GFS: virtio_net             27892  0

      T, [2013-05-02T12:41:07.795713 #13220] TRACE -- : GFS: virtio_scsi            18006  0

      T, [2013-05-02T12:41:07.798723 #13220] TRACE -- : GFS: virtio_blk             18325  1

      T, [2013-05-02T12:41:07.801536 #13220] TRACE -- : GFS: virtio_rng             13117  0

      T, [2013-05-02T12:41:07.865136 #13220] TRACE -- : GFS: virtio_balloon         13508  0

      T, [2013-05-02T12:41:07.869780 #13220] TRACE -- : GFS: virtio_mmio            13157  0

      T, [2013-05-02T12:41:07.872878 #13220] TRACE -- : GFS: sparse_keymap          13526  0

      T, [2013-05-02T12:41:07.876056 #13220] TRACE -- : GFS: rfkill                 21736  0

      T, [2013-05-02T12:41:07.878787 #13220] TRACE -- : GFS: sym53c8xx              76601  0

      T, [2013-05-02T12:41:07.882185 #13220] TRACE -- : GFS: scsi_transport_spi     30237  1 sym53c8xx

      T, [2013-05-02T12:41:07.885186 #13220] TRACE -- : GFS: crc8                   12750  0

      T, [2013-05-02T12:41:07.888184 #13220] TRACE -- : GFS: crc_ccitt              12613  0

      T, [2013-05-02T12:41:07.890839 #13220] TRACE -- : GFS: crc_itu_t              12613  0

      T, [2013-05-02T12:41:07.893473 #13220] TRACE -- : GFS: libcrc32c              12603  0

      T, [2013-05-02T12:41:08.109801 #13220] TRACE -- : GFS: Thu May  2 12:41:00 EDT 2013

      T, [2013-05-02T12:41:08.121044 #13220] TRACE -- : GFS: uptime:

      T, [2013-05-02T12:41:08.283117 #13220] TRACE -- : GFS: 16.09 0.88

      T, [2013-05-02T12:41:08.964743 #13220] TRACE -- : GFS: verbose daemon enabled

      T, [2013-05-02T12:41:08.969275 #13220] TRACE -- : GFS: linux commmand line: panic=1 console=ttyS0 udevtimeout=300 no_timer_check acpi=off printk.time=1 cgroup_disable=memory selinux=1 enforcing=0 guestfs_verbose=1 TERM=unknown n

      T, [2013-05-02T12:41:08.970603 #13220] TRACE -- : GFS: oapic

       

      T, [2013-05-02T12:41:09.733026 #13220] TRACE -- : GFS: udevadm settle

      T, [2013-05-02T12:41:10.001896 #13220] TRACE -- : GFS: udevd[54]: sender uid=65534, message ignored

       

      T, [2013-05-02T12:41:10.003555 #13220] TRACE -- : GFS:

      T, [2013-05-02T12:41:10.030779 #13220] TRACE -- : GFS: udevsettle

      T, [2013-05-02T12:41:10.049304 #13220] TRACE -- : GFS: udevsettle: No such file or directory

      T, [2013-05-02T12:41:10.118908 #13220] TRACE -- : GFS: [45538ms] appliance is up

      D, [2013-05-02T12:41:10.119152 #13220] DEBUG -- : GFS: launch = 0

      D, [2013-05-02T12:41:10.119276 #13220] DEBUG -- : GFS: list_devices

      T, [2013-05-02T12:41:10.124098 #13220] TRACE -- : GFS: guestfsd: main_loop: new request, len 0x28

      D, [2013-05-02T12:41:10.133314 #13220] DEBUG -- : GFS: list_devices = ["/dev/vda"]

      D, [2013-05-02T12:41:10.133579 #13220] DEBUG -- : GFS: list_partitions

      T, [2013-05-02T12:41:10.136355 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 7 (list_devices) took 0.01 seconds

      guestfsd: main_loop: new request, len 0x28

      D, [2013-05-02T12:41:10.209503 #13220] DEBUG -- : GFS: list_partitions = ["/dev/vda1", "/dev/vda2"]

      T, [2013-05-02T12:41:10.209746 #13220] TRACE -- : Mounting partitions...

      D, [2013-05-02T12:41:10.209900 #13220] DEBUG -- : GFS: list_partitions

      T, [2013-05-02T12:41:10.212103 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 8 (list_partitions) took 0.01 seconds

      guestfsd: main_loop: new request, len 0x28

      D, [2013-05-02T12:41:10.214408 #13220] DEBUG -- : GFS: list_partitions = ["/dev/vda1", "/dev/vda2"]

      D, [2013-05-02T12:41:10.219612 #13220] DEBUG -- : GFS: vfs_type "/dev/vda1"

      T, [2013-05-02T12:41:10.221009 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 8 (list_partitions) took 0.00 seconds

      T, [2013-05-02T12:41:10.223053 #13220] TRACE -- : GFS: guestfsd: main_loop: new request, len 0x38

      T, [2013-05-02T12:41:10.225675 #13220] TRACE -- : GFS: blkid -c /dev/null -o value -s TYPE /dev/vda1

      D, [2013-05-02T12:41:10.473473 #13220] DEBUG -- : GFS: vfs_type = ""

      D, [2013-05-02T12:41:10.473791 #13220] DEBUG -- : GFS: vfs_type "/dev/vda2"

      T, [2013-05-02T12:41:10.476996 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 198 (vfs_type) took 0.13 seconds

      guestfsd: main_loop: new request, len 0x38

      T, [2013-05-02T12:41:10.479242 #13220] TRACE -- : GFS: blkid -c /dev/null -o value -s TYPE /dev/vda2

      D, [2013-05-02T12:41:10.926133 #13220] DEBUG -- : GFS: vfs_type = "ext4"

      D, [2013-05-02T12:41:10.926375 #13220] DEBUG -- : GFS: vfs_type "/dev/vda2"

      T, [2013-05-02T12:41:10.929368 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 198 (vfs_type) took 0.17 seconds

      guest

      T, [2013-05-02T12:41:10.930598 #13220] TRACE -- : GFS: fsd: main_loop: new request, len 0x38

      T, [2013-05-02T12:41:10.932344 #13220] TRACE -- : GFS: blkid -c /dev/null -o value -s TYPE /dev/vda2

      D, [2013-05-02T12:41:11.292403 #13220] DEBUG -- : GFS: vfs_type = "ext4"

      T, [2013-05-02T12:41:11.292815 #13220] TRACE -- : Mounting /dev/vda2 partition to /...

      D, [2013-05-02T12:41:11.292929 #13220] DEBUG -- : GFS: mount_options "" "/dev/vda2" "/"

      T, [2013-05-02T12:41:11.295354 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 198 (vfs_type) took 0.18 seconds

      guestfsd: main_

      T, [2013-05-02T12:41:11.296581 #13220] TRACE -- : GFS: loop: new request, len 0x44

      T, [2013-05-02T12:41:11.305086 #13220] TRACE -- : GFS: mount -o  /dev/vda2 /sysroot/

      T, [2013-05-02T12:41:11.932311 #13220] TRACE -- : GFS: [   17.732460] EXT4-fs (vda2): mounted filesystem with ordered data mode. Opts: (null)

      D, [2013-05-02T12:41:12.055808 #13220] DEBUG -- : GFS: mount_options = 0

      D, [2013-05-02T12:41:12.061952 #13220] DEBUG -- : GFS: set_e2label "/dev/vda2" "79d3d2d4"

      T, [2013-05-02T12:41:12.063288 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 74 (mount_options) took 0.33 seconds

      T, [2013-05-02T12:41:12.126216 #13220] TRACE -- : GFS: guestfsd: main_loop: new request, len 0x44

      T, [2013-05-02T12:41:12.128929 #13220] TRACE -- : GFS: blkid -c /dev/null -o value -s TYPE /dev/vda2

      T, [2013-05-02T12:41:12.519844 #13220] TRACE -- : GFS: e2label /dev/vda2 79d3d2d4

      D, [2013-05-02T12:41:12.956901 #13220] DEBUG -- : GFS: set_e2label = 0

      T, [2013-05-02T12:41:12.957102 #13220] TRACE -- : Partition mounted.

      D, [2013-05-02T12:41:12.957235 #13220] DEBUG -- : GFS: exists "/etc/sysconfig/selinux"

      T, [2013-05-02T12:41:12.961100 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 80 (set_e2label) took 0.44 seconds

      T, [2013-05-02T12:41:13.026641 #13220] TRACE -- : GFS:

      guestfsd: main_loop: new request, len 0x44

      D, [2013-05-02T12:41:13.349594 #13220] DEBUG -- : GFS: exists = 1

      T, [2013-05-02T12:41:13.349809 #13220] TRACE -- : Loading SElinux policy...

      D, [2013-05-02T12:41:13.349959 #13220] DEBUG -- : GFS: aug_init "/" 32

      T, [2013-05-02T12:41:13.352055 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 36 (exists) took 0.15 seconds

      guestfsd: main_loop: new request, len 0x34

      D, [2013-05-02T12:41:34.871279 #13220] DEBUG -- : GFS: aug_init = 0

      D, [2013-05-02T12:41:34.871488 #13220] DEBUG -- : GFS: aug_rm "/augeas/load//incl[. != '/etc/sysconfig/selinux']"

      T, [2013-05-02T12:41:34.873402 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 16 (aug_init) took 9.16 seconds

      guestfsd: main_loop: new request, len 0x60

      D, [2013-05-02T12:41:34.893188 #13220] DEBUG -- : GFS: aug_rm = 294

      D, [2013-05-02T12:41:34.893399 #13220] DEBUG -- : GFS: aug_load

      T, [2013-05-02T12:41:34.895290 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 22 (aug_rm) took 0.02 seconds

      guestfsd: main_loop: new request, len 0x28

      D, [2013-05-02T12:41:35.616497 #13220] DEBUG -- : GFS: aug_load = 0

      D, [2013-05-02T12:41:35.616718 #13220] DEBUG -- : GFS: aug_get "/files/etc/sysconfig/selinux/SELINUX"

      T, [2013-05-02T12:41:35.708450 #13220] TRACE -- : GFS: guestfsd: main_loop: proc 27 (aug_load) took 0.30 seconds

      guestfsd: main_loop:

      T, [2013-05-02T12:41:35.710549 #13220] TRACE -- : GFS:  new request, len 0x50

      T, [2013-05-02T12:41:35.712785 #13220] TRACE -- : GFS: guestfsd: error: no matching node

      D, [2013-05-02T12:41:35.712928 #13220] DEBUG -- : GFS: aug_get = NULL (error)

      F, [2013-05-02T12:41:35.713358 #13220] FATAL -- : Guestfs::Error: aug_get: no matching node

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:219:in `aug_get'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:219:in `load_selinux_policy'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:202:in `execute'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:174:in `block in customize'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:123:in `block (2 levels) in initialize_guestfs'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:169:in `prepare_guestfs'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:123:in `block in initialize_guestfs'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:77:in `log_callback'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:123:in `initialize_guestfs'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb:173:in `customize'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/image-helper.rb:129:in `customize'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/plugins/os/rpm-based/rpm-based-os-plugin.rb:76:in `build_with_appliance_creator'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/plugins/os/fedora/fedora-plugin.rb:53:in `execute'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/plugins/base-plugin.rb:172:in `run'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:184:in `execute_plugin'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:200:in `block in execute_with_userchange'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/util/permissions/user-switcher.rb:27:in `call'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/util/permissions/user-switcher.rb:27:in `change_user'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:199:in `execute_with_userchange'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:127:in `block in execute_plugin_chain'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:125:in `each'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:125:in `execute_plugin_chain'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/appliance.rb:164:in `create'

      /usr/share/gems/gems/boxgrinder-build-0.10.4/bin/boxgrinder-build:203:in `<top (required)>'

      /usr/bin/boxgrinder-build:23:in `load'

      /usr/bin/boxgrinder-build:23:in `<main>'

       

      I searched the web for similar errors and all I could find was this from the libguestfs mailing list last January:

       

      https://www.redhat.com/archives/libguestfs/2013-January/msg00079.html

       

      The libguestfs people seem to think that this is a boxgrinder bug.

       

      The file /etc/selinux/config file on my appliance exists and has SELINUX=disabled

       

      I'm happy to attach any further files to help diagnose the problem.


        • 1. Re: F17 boxgrinder-meta appliance doesn't seem to be able to grind an F17 boxgrinder-meta appliance
          sweeney

          I mounted the partially built appliance disk image, and the /etc/sysconfig/selinux file in the target file system there is as follows:

           

          [root@localhost ~]# cd /mnt/boxgrinder-meta

          [root@localhost boxgrinder-meta]# cat etc/sysconfig/selinux

          # This file controls the state of SELinux on the system.

          # SELINUX= can take one of these three values:

          #       enforcing - SELinux security policy is enforced.

          #       permissive - SELinux prints warnings instead of enforcing.

          #       disabled - SELinux is fully disabled.

          SELINUX=permissive

          # SELINUXTYPE= type of policy in use. Possible values are:

          #       targeted - Only targeted network daemons are protected.

          #       strict - Full SELinux protection.

          SELINUXTYPE=targeted

          [root@localhost boxgrinder-meta]#

          • 2. Re: F17 boxgrinder-meta appliance doesn't seem to be able to grind an F17 boxgrinder-meta appliance
            sweeney

            So, I loaded the augeas command line tool on both the F15 boxgrinder-meta in our data center and on the F17 node in AWS, and the problem seems to be that /etc/sysconfig/selinux is no longer in the Shellvars lens in the F17 version.

             

            F15:

            [root@boxgrinder2 ~]# augtool

            augtool> print /augeas/load//incl[. = '/etc/sysconfig/selinux']

            /augeas/load/Shellvars/incl[15] = "/etc/sysconfig/selinux"

            augtool>

             

            F17:

            augtool> print /augeas/load//incl[. = '/etc/sysconfig/selinux']

            augtool>

             

            So, F15 can grind an F17 appliance, but F17 can't.

            • 3. Re: F17 boxgrinder-meta appliance doesn't seem to be able to grind an F17 boxgrinder-meta appliance
              sweeney

              It turns out that on both F15 and F17, the file /etc/sysconfig/selinux is in fact a symbolic link:

               

              [root@boxgrinder2 ~]# ls -li /etc/selinux/config /etc/sysconfig/selinux

              400501 -rw-r--r--. 1 root root 449 Feb 17  2012 /etc/selinux/config

              400502 lrwxrwxrwx. 1 root root  17 Feb 17  2012 /etc/sysconfig/selinux -> ../selinux/config

              [root@boxgrinder2 ~]#

               

              This appears to also be true on CentOS5 and CentOS6 and presumably other headgear distros

               

              The Augeas Shellvars lens in F15 includes /etc/sysconfig/selinux, whereas the F17 version has only /etc/selinux/config

               

              If I change the three occurrences of etc/sysconfig/selinux to etc/selinux/config in /usr/share/gems/gems/boxgrinder-build-0.10.4/lib/boxgrinder-build/helpers/guestfs-helper.rb, then the (hacked) F17 boxgrinder-meta appliance can correctly build an F17 boxgrinder-meta appliance.  This hack would, however, fail if run on F15, with the same problem I was experiencing earlier.