Dependency issue on rubygems?

[starksm64]

I'm working on building a new appliance using my own RPM in a local repository, and on the first try to create the appliance I see this error:

I, [2010-08-26T10:13:22.720716 #27585]  INFO -- : Building hostmanager appliance...
E, [2010-08-26T10:13:41.610360 #27585] ERROR -- : /usr/lib/ruby/gems/1.8/gems/boxgrinder-core-0.0.22/lib/boxgrinder-core/helpers/exec-helper.rb:62:in `execute'
/usr/lib/ruby/gems/1.8/gems/boxgrinder-build-rpm-based-os-plugin-0.0.3/lib/boxgrinder-build-rpm-based-os-plugin/rpm-based-os-plugin.rb:43:in `build_with_appliance_creator'
/usr/lib/ruby/gems/1.8/gems/boxgrinder-build-fedora-os-plugin-0.0.2/lib/boxgrinder-build-fedora-os-plugin/fedora-plugin.rb:61:in `execute'
/usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.5.1/lib/boxgrinder-build/plugins/base-plugin.rb:90:in `run'
/usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.5.1/lib/boxgrinder-build/appliance.rb:71:in `execute_os_plugin'
/usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.5.1/lib/boxgrinder-build/appliance.rb:55:in `create'
/usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.5.1/bin/boxgrinder-build:88
/usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
/usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
/usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
/usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
/usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/runner.rb:74:in `run!'
/usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/delegates.rb:7:in `run!'
/usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/import.rb:10
/usr/bin/boxgrinder-build:19
E, [2010-08-26T10:13:41.610561 #27585] ERROR -- : An error occurred while executing command: 'appliance-creator -d -v -t build/appliances/i386/fedora/13/hostmanager/fedora-plugin/tmp --cache=/var/cache/boxgrinder/rpms-cache/i386/fedora/13 --config build/appliances/i386/fedora/13/hostmanager/fedora-plugin/hostmanager.ks -o build/appliances/i386/fedora/13/hostmanager/fedora-plugin/tmp --name hostmanager --vmem 512 --vcpu 1', process exited with wrong exit status: 1
/usr/lib/ruby/gems/1.8/gems/boxgrinder-core-0.0.22/lib/boxgrinder-core/helpers/exec-helper.rb:68:in `execute': An error occurred while executing command: 'appliance-creator -d -v -t build/appliances/i386/fedora/13/hostmanager/fedora-plugin/tmp --cache=/var/cache/boxgrinder/rpms-cache/i386/fedora/13 --config build/appliances/i386/fedora/13/hostmanager/fedora-plugin/hostmanager.ks -o build/appliances/i386/fedora/13/hostmanager/fedora-plugin/tmp --name hostmanager --vmem 512 --vcpu 1', process exited with wrong exit status: 1 (RuntimeError)
    from /usr/lib/ruby/gems/1.8/gems/boxgrinder-build-rpm-based-os-plugin-0.0.3/lib/boxgrinder-build-rpm-based-os-plugin/rpm-based-os-plugin.rb:43:in `build_with_appliance_creator'
    from /usr/lib/ruby/gems/1.8/gems/boxgrinder-build-fedora-os-plugin-0.0.2/lib/boxgrinder-build-fedora-os-plugin/fedora-plugin.rb:61:in `execute'
    from /usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.5.1/lib/boxgrinder-build/plugins/base-plugin.rb:90:in `run'
    from /usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.5.1/lib/boxgrinder-build/appliance.rb:71:in `execute_os_plugin'
    from /usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.5.1/lib/boxgrinder-build/appliance.rb:55:in `create'
    from /usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.5.1/bin/boxgrinder-build:88
    from /usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
    from /usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
    from /usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
    from /usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
    from /usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/runner.rb:74:in `run!'
    from /usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/delegates.rb:7:in `run!'
    from /usr/lib/ruby/gems/1.8/gems/commander-4.0.3/lib/commander/import.rb:10
    from /usr/bin/boxgrinder-build:19
[starksm@fedora13 BoxGrinder]$

Since I could not tell what the problem is from this, I ran the command that failed and it showed:

[starksm@fedora13 BoxGrinder]$ sudo appliance-creator -d -v -t build/appliances/i386/fedora/13/hostmanager/fedora-plugin/tmp --cache=/var/cache/boxgrinder/rpms-cache/i386/fedora/13 --config build/appliances/i386/fedora/13/hostmanager/fedora-plugin/hostmanager.ks -o build/appliances/i386/fedora/13/hostmanager/fedora-plugin/tmp --name hostmanager --vmem 512 --vcpu 1
[sudo] password for starksm: 
Adding disk sda as /home/starksm/Dev/BoxGrinder/build/appliances/i386/fedora/13/hostmanager/fedora-plugin/tmp/imgcreate-uijOsv/tmp-dFUlsN/hostmanager-sda.raw
Extending sparse file /home/starksm/Dev/BoxGrinder/build/appliances/i386/fedora/13/hostmanager/fedora-plugin/tmp/imgcreate-uijOsv/tmp-dFUlsN/hostmanager-sda.raw to 2147483648
Losetup add /dev/loop0 mapping to /home/starksm/Dev/BoxGrinder/build/appliances/i386/fedora/13/hostmanager/fedora-plugin/tmp/imgcreate-uijOsv/tmp-dFUlsN/hostmanager-sda.raw
Formatting disks
Initializing partition table for /dev/loop0
Assigning partitions to disks
Assigned / to sda1 at 0 at size 2048
Creating partitions
Add primary part at 0 of size 2048
Warning: The resulting partition is not properly aligned for best performance.
Dev loop0p1: /dev/loop01 -> /dev/mapper/loop0p1
Adding partx mapping for /dev/loop0
['/']
Formating ext3 filesystem on /dev/loop01
mke2fs 1.41.10 (10-Feb-2009)
Filesystem label=/
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
125184 inodes, 500000 blocks
5000 blocks (1.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=515899392
16 block groups
32768 blocks per group, 32768 fragments per group
7824 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912

Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
Tuning filesystem on /dev/loop01
tune2fs 1.41.10 (10-Feb-2009)
Setting maximal mount count to -1
Setting interval between checks to 0 seconds
Mounting /dev/loop01 at /home/starksm/Dev/BoxGrinder/build/appliances/i386/fedora/13/hostmanager/fedora-plugin/tmp/imgcreate-uijOsv/install_root/
Writing mkinitrd config /home/starksm/Dev/BoxGrinder/build/appliances/i386/fedora/13/hostmanager/fedora-plugin/tmp/imgcreate-uijOsv/install_root/etc/sysconfig/mkinitrd
Retrieving http://repo.boxgrinder.org/cirras/packages/fedora/13/RPMS/i386/repodata/repomd.xml ...OK
Retrieving http://repo.boxgrinder.org/cirras/packages/fedora/13/RPMS/noarch/repodata/repomd.xml ...OK
Retrieving http://fedora.secsup.org/linux/releases/13/Everything/i386/os/repodata/repomd.xml ...OK
Retrieving http://astromirror.uchicago.edu/fedora/linux/updates/13/i386/repodata/repomd.xml ...OK
Retrieving http://astromirror.uchicago.edu/fedora/linux/updates/13/i386/repodata/b29c464ed4e70342a8fef7b15df2e2132f2bd7e071951b73d38c3bacae4edcab-primary.sqlite.bz2 ...OK
Retrieving file:///opt/repo/RPMS/i386/repodata/repomd.xml ...OK
Retrieving file:///opt/repo/RPMS/noarch/repodata/repomd.xml ...OK
/usr/lib/python2.6/site-packages/imgcreate/errors.py:40: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  return str(self.message)
Unable to create appliance : Failed to build transaction : rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch requires rubygem(json) >= 1.4.6
rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch requires rubygem(progressbar) >= 0.9.0
rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch requires rubygem(net-sftp) >= 2.0.4
rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch requires rubygem(bluepill) >= 0.0.39
rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch requires rubygem(boxgrinder-core) >= 0.0.5
rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch requires rubygem(amazon-ec2) >= 0.9.6
rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch requires rubygem(aws-s3) >= 0.6.2
Unmounting directory /home/starksm/Dev/BoxGrinder/build/appliances/i386/fedora/13/hostmanager/fedora-plugin/tmp/imgcreate-uijOsv/install_root/
Removing compat symlinks
Unmapping /dev/loop0
Losetup remove /dev/loop0

So apparently I'm missing several rubygems, but I do have all of thoses listed:

[starksm@fedora13 jboss-hostmanager]$ for g in json progressbar net-sftp bluepill boxgrinder-core amazon-ec2 aws-s3
> do
> gem list $g
> done

*** LOCAL GEMS ***

json (1.4.6)

*** LOCAL GEMS ***

progressbar (0.9.0)

*** LOCAL GEMS ***

net-sftp (2.0.5)

*** LOCAL GEMS ***

bluepill (0.0.42, 0.0.39)

*** LOCAL GEMS ***

boxgrinder-core (0.0.22, 0.0.21)

*** LOCAL GEMS ***

amazon-ec2 (0.9.15)

*** LOCAL GEMS ***

aws-s3 (0.6.2)

What am I missing here?

[starksm64]

The spec file that create the rpm is:

# Generated from jboss-aws-hostmanager-0.1.0.gem by gem2rpm -*- rpm-spec -*-
%global ruby_sitelib %(ruby -rrbconfig -e "puts Config::CONFIG['sitelibdir']")
%global gemdir %(ruby -rubygems -e 'puts Gem::dir' 2>/dev/null)
%global gemname jboss-aws-hostmanager
%global geminstdir %{gemdir}/gems/%{gemname}-%{version}
%global jbossaws_home /opt/jbossawsmac

Summary: A HostManager implementation for AWS
Name: rubygem-%{gemname}
Version: 0.1.0
Release: 1%{?dist}
Group: Development/Languages
License: GPLv2+ or Ruby
URL: http://www.jboss.org/stormgrind/projects/boxgrinder.html
Source0: %{gemname}-%{version}.gem
Requires: rubygems
Requires: rubygem(boxgrinder-core) >= 0.0.5
Requires: rubygem(aws-s3) >= 0.6.2
Requires: rubygem(amazon-ec2) >= 0.9.6
Requires: rubygem(net-sftp) >= 2.0.4
Requires: rubygem(net-ssh) >= 2.0.20
Requires: rubygem(rake) >= 0.8.7
Requires: rubygem(progressbar) >= 0.9.0
Requires: rubygem(bluepill) >= 0.0.39
Requires: rubygem(json) >= 1.4.6
BuildRequires: rubygems
BuildArch: noarch
Provides: rubygem(%{gemname}) = %{version}

%description
The AWS HostManager for JBoss EE6 web applications


%prep

%build

%install
rm -rf %{buildroot}
mkdir -p %{buildroot}%{gemdir}
gem install --local --install-dir %{buildroot}%{gemdir} \
            --force --rdoc %{SOURCE0}
# Copy
echo install -d -m 755 $RPM_BUILD_ROOT/../%{jbossaws_home}
install -d -m 755 $RPM_BUILD_ROOT/../%{jbossaws_home}
cp -pr %{buildroot}%{gemdir}/gems/%{gemname}-%{version}/config $RPM_BUILD_ROOT/../%{jbossaws_home}

mkdir -p %{buildroot}/%{_bindir}
mv %{buildroot}%{gemdir}/bin/* %{buildroot}/%{_bindir}
rmdir %{buildroot}%{gemdir}/bin
find %{buildroot}%{geminstdir}/bin -type f | xargs chmod a+x

%clean
#rm -rf %{buildroot}/%{jbossaws_home}

%files
%defattr(-, root, root, -)
%{_bindir}/jboss-hostmanager
%{gemdir}/gems/%{gemname}-%{version}/
%doc %{gemdir}/doc/%{gemname}-%{version}
%doc %{geminstdir}/README.txt
%{gemdir}/cache/%{gemname}-%{version}.gem
%{gemdir}/specifications/%{gemname}-%{version}.gemspec


%changelog
* Wed Aug 25 2010 Scott M Stark <[e-mail]> - 0.1.0-1
- Initial package

None of these rubygem(...) requirements are installed as rpm packages, so there seems to be a disconnect between gem2rpm and how the rubygems requirements are resolved.

[starksm64]

Just trying to install the rpm also fails with the missing dependencies. Clearly I'm missing how the dependencies added by gem2rpm are supposed to be resolved.

[starksm@fedora13 jboss-hostmanager]$ sudo rpm -ip /opt/repo/RPMS/noarch/rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch.rpm 
error: Failed dependencies:
    rubygem(amazon-ec2) >= 0.9.6 is needed by rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch
    rubygem(aws-s3) >= 0.6.2 is needed by rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch
    rubygem(bluepill) >= 0.0.39 is needed by rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch
    rubygem(boxgrinder-core) >= 0.0.5 is needed by rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch
    rubygem(json) >= 1.4.6 is needed by rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch
    rubygem(net-sftp) >= 2.0.4 is needed by rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch
    rubygem(net-ssh) >= 2.0.20 is needed by rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch
    rubygem(progressbar) >= 0.9.0 is needed by rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch
    rubygem(rake) >= 0.8.7 is needed by rubygem-jboss-aws-hostmanager-0.1.0-1.fc13.noarch

[starksm64]

I'm working around this by just having my rubygem-jboss-aws-hostmanager rpm install its required gems in the %pre phase. This is the behavior I want anyway. It would be nice if there was a way to have dependencies on gems specfied in the appliance file and have the boxgrinder build take care of this. Should I create a feature request for this?

%pre
# Install the required gems
gem install boxgrinder-core --version '>= 0.0.5'
gem install aws-s3 --version '>= 0.6.2'
gem install amazon-ec2 --version '>= 0.9.6'
gem install net-sftp --version ' >= 2.0.4'
gem install net-ssh --version ' >= 2.0.20'
gem install rake --version ' >= 0.8.7'
gem install progressbar --version ' >= 0.9.0'
gem install bluepill --version ' >= 0.0.39'
gem install json --version '>= 1.4.6'

[goldmann]

Hi Scott,

Sorry for late response – I haven't received email notifications.

The way it should be done is to add required GEM's as packaged RPMs to the operating system you use (Fedora for example). This what you proposed here is a simple workaround, which works, but the RPM doesn't meets Fedora package guidelines

I How To Package And Include In Fedora A Gem? describing the right process of adding gems to Fedora.

Btw. I see you require BoxGrinder gems. I think this is not required in your case. Do you use net-sftp package or boxgrinder-core in your software?

--Marek

[starksm64]

So, the wiki appears to imply that the gems need to have been installed as rpm packages in order for them to be resolved via Requires: rubygem(...) to be resolved?

I don't need to depend on boxgrinder gems, it was just something that leaked in there, and this is true for net-sftp as well.

Building a fedora based image that relies on ruby gems should not require having to get the gems into fedora proper. That is unlikely to be possible in general.

What I would like to see if better support in the boxgrinder appliance description for ruby gems. Can we use the same syntax of rubygems(...) so that I can write an appliance file like:

[starksm@fedora13 BoxGrinder]$ cat appliances/hostmanager.appl
name: hostmanager
version: 1
release: 0
summary: JBoss AWS Hostmanager appliance with JBoss AS 6
hardware:
  memory: 512
  partitions:
    "/":
      size: 2
appliances:
  - cirras
packages:
  includes:
    - jboss-as6
    - java-1.6.0-openjdk
    - rubygem-jboss-aws-hostmanager

    - rubygem(amazon-ec2) >= 0.9.6

    - rubygem(bluepill) >= 0.0.39

repos:
  - name: "local-noarch"
    baseurl: "file:///opt/repo/RPMS/noarch"
    ephemeral: true
  - name: "local-#ARCH#"
    baseurl: "file:///opt/repo/RPMS/#ARCH#"
    ephemeral: true

and have the amazon-ec2 and bluepill gems installed via 'gem install' during the build of the image?

[goldmann]

Hi Scott,

Scott Stark wrote:

So, the wiki appears to imply that the gems need to have been installed as rpm packages in order for them to be resolved via Requires: rubygem(...) to be resolved?

Yes, that's true. RPM is the way software is packaged in Fedora. So, even if we have a package solution like gem, we need still to repackage it into RPM.

Building a fedora based image that relies on ruby gems should not require having to get the gems into fedora proper. That is unlikely to be possible in general.

Unfortunately this is the way to go if you want to release your software. If you want it to have it just working and wants to support it on your own - you're free to use one of available hacks, see below.

What I would like to see if better support in the boxgrinder appliance description for ruby gems. Can we use the same syntax of rubygems(...) so that I can write an appliance file like:

[starksm@fedora13 BoxGrinder]$ cat appliances/hostmanager.appl
name: hostmanager
version: 1
release: 0
summary: JBoss AWS Hostmanager appliance with JBoss AS 6
hardware:
  memory: 512
  partitions:
    "/":
      size: 2
appliances:
  - cirras
packages:
  includes:
    - jboss-as6
    - java-1.6.0-openjdk
    - rubygem-jboss-aws-hostmanager

    - rubygem(amazon-ec2) >= 0.9.6

    - rubygem(bluepill) >= 0.0.39

repos:
  - name: "local-noarch"
    baseurl: "file:///opt/repo/RPMS/noarch"
    ephemeral: true
  - name: "local-#ARCH#"
    baseurl: "file:///opt/repo/RPMS/#ARCH#"
    ephemeral: true

and have the amazon-ec2 and bluepill gems installed via 'gem install' during the build of the image?

Proper names for packaged gems are: rubygem-amazon-ec2, rubygem-bluepill. Still you need to have gems as RPM's – packages list is directly translated into kickstart RPM list.

I think the best way to build your appliance is to build the required RPM's from gems and place them in your local repo and then include only one in packages list: rubygem-jboss-aws-hostmanager. Everything else should be pulled from your local repo.

--Marek

[starksm64]

Marek Goldmann wrote:

Proper names for packaged gems are: rubygem-amazon-ec2, rubygem-bluepill. Still you need to have gems as RPM's – packages list is directly translated into kickstart RPM list.

I think the best way to build your appliance is to build the required RPM's from gems and place them in your local repo and then include only one in packages list: rubygem-jboss-aws-hostmanager. Everything else should be pulled from your local repo.

--Marek

Ok, so we are relying on kickstart files to build the image. I think the simplest solution is just to run 'gem install ...' in my spec %pre section of the rpm. I don't see much benefit to repacking the gems as rpms into a local repository when during the image build I can call out to the gem management tool when my hostmanager rpm is being installed as part of the kickstart script.

[goldmann]

Scott Stark wrote:

Ok, so we are relying on kickstart files to build the image. I think the simplest solution is just to run 'gem install ...' in my spec %pre section of the rpm. I don't see much benefit to repacking the gems as rpms into a local repository when during the image build I can call out to the gem management tool when my hostmanager rpm is being installed as part of the kickstart script.

Or this way - sure. It is just not so clean

--Marek

[starksm64]

What do you see as the main problem with that approach? It would seem cleaner from the perspective that I'm using the language specific package management feature vs relying on the operating system package manager, which requires an extra step. I would have a similar view of wanting to use maven or ivy to install java packages rather than having to bundle up java based rpms.

Ideally rpm would have better integration for language specific repositories so that you could point to a rubygems or maven repository in your spec file, and then have the Requires: ... use a rubygems(...) or maven(...) package reference and have the installed in the fedora repository. I'll bring this up on the arch list to see what the feedback is.

[goldmann]

Scott,

I think you misunderstood me. I am +100 for directly installing gems using gem command, but it causes sometimes problems on RPM based systems. One is obvious – we have now two package managers – wich is more important? And second thing – sometimes a gem has a dependency which isn't a gem but a RPM package – how will you enforce resolving this dependency? I have this problem with BoxGrinder currenty – and I'll solve it by packaging the gem into RPM. Then you just run

yum install boxgrinder-build-fedora-os-plugins

and you'll have BG ready to run.

Ideally rpm would have better integration for language specific repositories so that you could point to a rubygems or maven repository in your spec file, and then have the Requires: ... use a rubygems(...) or maven(...) package reference and have the installed in the fedora repository. I'll bring this up on the arch list to see what the feedback is.

Yes, I completely agree. If RPM could be a proxy for language specific package managers – would be great!

--Marek