Documentation and Controlling image creation

[tiradani]

Hello,

Just getting started with trying out BoxGrinder.  I am quite impressed that I was able to install and create a simple VM image with no problems.  I am now trying to create images for specific use cases and am running into some issues that I hope someone can help with.

1) As of version 0.8 the "includes:" section was removed (and by extension "excludes:" was removed as well?).  How does one go about excluding specific packages or at least go back and remove all unnecessary packages?  I am creating a CentOS image.  I specify what should be a minimal set of packages (at least they are if I use a kickstart file to install).  When I follow the install process with BoxGrinder, I see nearly twice as many packages being installed compared to a straight kickstart install.  By the way, the docs still say to use "includes:".

2) I see in the documentation how to specify custom repositories.  It appears that the repos I define are being ignored and it is always looking at some boxgrinder repos.  Is that simply a misunderstanding of the log messages on my part, or is BoxGrinder hard coded to look at certain repos? (This is for a CentOS appliance.) 

3) How do I specify complex sequences in the post section rather than one-liner commands?  Example of what I am looking to do:

cat > /path/to/some/file << EOF

echo "creating some script"

EOF

4) Is there any documentation detailing all possible options in the YAML appliance definition file?

I also have one feature request.  Could the error message associated with each exception (or whatever ruby calls them) be logged somewhere?  What currently happens is that an exception is raised and caught at the next level up.  The error message is discarded and another exception is raised with a new error message.  This repeats until it hits the top level and what the user sees is some generic error message and a stack trace.  I ran into this because I had one line with a syntax error in my appliance definition file.  I only found out what the actual problem was because I went into the code and inserted logging just prior to each raise so I could see what the actual error was.  (This is my first experience with both ruby and yaml.)

Thanks in advance for the help!

[goldmann]

tiradani wrote:

Hello,

Just getting started with trying out BoxGrinder.  I am quite impressed that I was able to install and create a simple VM image with no problems.  I am now trying to create images for specific use cases and am running into some issues that I hope someone can help with.

Great to hear this!

1) As of version 0.8 the "includes:" section was removed (and by extension "excludes:" was removed as well?).  How does one go about excluding specific packages or at least go back and remove all unnecessary packages?  I am creating a CentOS image.  I specify what should be a minimal set of packages (at least they are if I use a kickstart file to install).  When I follow the install process with BoxGrinder, I see nearly twice as many packages being installed compared to a straight kickstart install.  By the way, the docs still say to use "includes:".

Yes, includes and excludes was removed. The idea is that you need to choose the minimal package set. You cannot break the dependencies because you would end with a not working appliance. To be honest, don't worry much about the size - it really doesn't matter. By default all @core packages are added and this is the JEOS for Fedora or CentOS. You may want to add more packages if you wish.

Yes, I can imagine you could have smaller package set, but our goal is not to create minimal images, but usable images. As I said above - @core package group should be the minimal OS. It was created with this in mind.

Docs are moved here: http://boxgrinder.org/tutorials/ I'm working on disabling/redirecting old docs.

2) I see in the documentation how to specify custom repositories.  It appears that the repos I define are being ignored and it is always looking at some boxgrinder repos.  Is that simply a misunderstanding of the log messages on my part, or is BoxGrinder hard coded to look at certain repos? (This is for a CentOS appliance.)

Only default CentOS repos are hardcoded. No BoxGrinder-related repos are added. Could you please provide the log so I can tell you more?

3) How do I specify complex sequences in the post section rather than one-liner commands?  Example of what I am looking to do:

cat > /path/to/some/file << EOF

echo "creating some script"

EOF

There are two ways of doing it:

1. Create RPM with your configuration files and include it into appliance.
2. Use specific YAML structure, like this:

post:
  base:
    - |
      echo "hello"
      echo 'world'

4) Is there any documentation detailing all possible options in the YAML appliance definition file?

This may help you: http://boxgrinder.org/tutorials/appliance-definition/ See the first note.

I also have one feature request.  Could the error message associated with each exception (or whatever ruby calls them) be logged somewhere?  What currently happens is that an exception is raised and caught at the next level up.  The error message is discarded and another exception is raised with a new error message.  This repeats until it hits the top level and what the user sees is some generic error message and a stack trace.  I ran into this because I had one line with a syntax error in my appliance definition file.  I only found out what the actual problem was because I went into the code and inserted logging just prior to each raise so I could see what the actual error was.  (This is my first experience with both ruby and yaml.)

Yes, I'm aware of this issue, this is the only place where it could happen. Proper validation of appliance definitions will be added soon, with a (hopefully) detailed info where something was messed up

--Marek

[tiradani]

Hi Marek,

Thank you very much for the quick response!

1) As of version 0.8 the "includes:" section was removed (and by extension "excludes:" was removed as well?).  How does one go about excluding specific packages or at least go back and remove all unnecessary packages?  I am creating a CentOS image.  I specify what should be a minimal set of packages (at least they are if I use a kickstart file to install).  When I follow the install process with BoxGrinder, I see nearly twice as many packages being installed compared to a straight kickstart install.  By the way, the docs still say to use "includes:".

Yes, includes and excludes was removed. The idea is that you need to choose the minimal package set. You cannot break the dependencies because you would end with a not working appliance. To be honest, don't worry much about the size - it really doesn't matter. By default all @core packages are added and this is the JEOS for Fedora or CentOS. You may want to add more packages if you wish.

Yes, I can imagine you could have smaller package set, but our goal is not to create minimal images, but usable images. As I said above - @core package group should be the minimal OS. It was created with this in mind.

Docs are moved here: http://boxgrinder.org/tutorials/ I'm working on disabling/redirecting old docs.

Ok, I can work around this then.  @Core really does add more than needed.  The project I am working on right now benefits greatly from smaller image sizes, but it isn't a critical requirement.  I appreciate the information.

2) I see in the documentation how to specify custom repositories.  It appears that the repos I define are being ignored and it is always looking at some boxgrinder repos.  Is that simply a misunderstanding of the log messages on my part, or is BoxGrinder hard coded to look at certain repos? (This is for a CentOS appliance.)

Only default CentOS repos are hardcoded. No BoxGrinder-related repos are added. Could you please provide the log so I can tell you more?

I had turned trace on so the log file is huge.  Here are a couple of lines that prompted my question:

I, [2011-02-21T14:49:55.804903 #20073]  INFO -- : Resolving packages added to centos5-min appliance definition file...

D, [2011-02-21T14:49:55.811901 #20073] DEBUG -- : Querying package database...

D, [2011-02-21T14:49:55.812144 #20073] DEBUG -- : Executing command: 'repoquery --quiet --disablerepo=* --enablerepo=boxgrinder-centos-5-base,boxgrinder-centos-5-updates,boxgrinder-base,boxgrinder-Updates -c 'build/appliances/x86_64/centos/5/centos5-min/centos-plugin/tmp/yum.conf' list available audit-libs basesystem bash beecrypt bzip2-libs centos-release centos-release-notes chkconfig coreutils cpio cracklib cracklib-dicts db4 device-mapper device-mapper-event device-mapper-multipath dhclient diffutils dmraid e2fsprogs e2fsprogs-libs elfutils-libelf ethtool expat filesystem findutils gawk gdbm glib2 glibc glibc-common grep grub gzip info initscripts iproute iputils kernel keyutils-libs kpartx krb5-libs less libacl libattr libcap libgcc libselinux libsepol libstdc++ libsysfs libtermcap lvm2 m2crypto MAKEDEV mcstrans mkinitrd mktemp module-init-tools nash ncurses net-tools openssl pam passwd pcre popt procps psmisc python readline redhat-logos rootfiles rpm rpm-libs sed setup shadow-utils sqlite sysklogd SysVinit tar termcap tzdata udev util-linux vim-minimal yum zlib curl system-config-securitylevel-tui --nevra --archlist=x86_64,noarch'

It is the --enablerepo arguments that caused the question.  I had defined a different set of repos to install from and none of them are listed.  Is there a way to override this behavior?

Thank you for the very helpful information.

[goldmann]

This step checks if all added packages will be resolved in specified repositories. We create a local YUM configuration file where we store the selected repos with a "boxgrinder-" prefix to avoid accidental interference with you system wide repos. Don't worry about this, this doesn't mean we're adding other repos or something.

--Marek