11 Replies Latest reply on Jul 1, 2011 5:24 PM by msavy

    Cannot deliver Fedora 15 appliance to EC2 with EBS or S3 plugins (eu-west-1 availability zone)

    unknow

      Hi,

       

      I've recently been trying BoxGrinder because I need to create a new Fedora-powered server on Amazon EC2 for each new release of Fedora. I'm currently trying to create an appliance for Fedora 15.

       

      Although the appliance building process seems to work with no errors, I was unable to deliver the appliance to the cloud. Note that I need the appliance to run in the eu-west availability zone.

       

      Can anyone please help me solve this problem or direct me to relevant documentation?

       

      EBS delivery plugin

       

      I have successfully launched a meta appliance in the us-east zone and created an EBS-based AMI in that zone. However, if I try to specify the availability_zone parameter in the configuration, I get the following error:

       

      F, [2011-06-05T10:45:45.642256 #19926] FATAL -- : RuntimeError: You selected eu-west-1b avaibility zone, but your instance is running in us-east-1d zone. Please change avaibility zone in plugin configuration file to us-east-1d (see http://boxgrinder.org/tutorials/boxgrinder-build-plugins/#EBS_Delivery_Plugin) or use another instance in eu-west-1b zone to create your EBS AMI.

       

      I'm not sure I understand the use of this parameter if it can't be different than the availability zone the instance is running in.

       

      There are no meta appliances provided for other zones either. Is there an easy way to create an EBS-backed AMI in another zone that us-east?

       

      S3 delivery plugin

       

      Since I've already converted instance-store AMIs to EBS-backed AMIs, I also tried to deliver the appliance with the S3 plugin. However, I always get this error:

       

      I, [2011-06-05T10:56:14.471182 #20044]  INFO -- : Image converted to EC2 format.

      T, [2011-06-05T10:56:14.471416 #20044] TRACE -- : Moving 'build/appliances/x86_64/fedora/15/my_server-f15/ec2-plugin/tmp/my_server-f15.ec2' deliverable to target destination 'build/appliances/x86_64/fedora/15/my_server-f15/ec2-plugin/my_server-f15.ec2'...

      D, [2011-06-05T10:56:14.472618 #20044] DEBUG -- : Platform plugin executed.

      T, [2011-06-05T10:56:14.474719 #20044] TRACE -- : Checking if 'my_server-f15/fedora/15/2.0/x86_64/my_server-f15.ec2.manifest.xml' path exists in my_server...

      T, [2011-06-05T10:56:14.474926 #20044] TRACE -- : Path doesn't exist!

      F, [2011-06-05T10:56:14.475447 #20044] FATAL -- : Errno::ENOENT: No such file or directory - build/appliances/x86_64/fedora/15/my_server-f15/s3-plugin/ami/my_server-f15.ec2.manifest.xml

       

      The problem seemed similar to this post (http://community.jboss.org/message/558438), but trying the proposed solution did not change anything. I have ensured that the AMI tools are installed and working properly, but the error remains.

       

      My appliance definition file

       

      name: my_server

      summary: my_server AMI with Fedora 15

      version: 2

      os:

        name: fedora

        version: 15

      hardware:

        partitions:

          "/":

            size: 10

       

      My .boxgrinder/config file

       

      plugins:

        s3:

          access_key: YYYY                        # (required)

          secret_access_key: YYYY          # (required)

          bucket: my_bucket                           # (required)

          account_number: YYYY                    # (required)

          cert_file: /root/.ec2/cert-YYYY.pem                  # required only for ami type

          key_file: /root/.ec2/pk-YYYY.pem                     # required only for ami type

          region: eu-west-1b

          availability_zone: eu-west-1b

        ebs:

          access_key: YYYY                        # (required)

          secret_access_key: YYYY          # (required)

          account_number: YYYY                    # (required)

          availability_zone: eu-west-1b

          region: eu-west-1b

       

      Title updated by unknow

        • 1. Re: Cannot deliver Fedora 15 appliance to EC2 with EBS or S3 plugins
          goldmann
          EBS delivery plugin

           

          I have successfully launched a meta appliance in the us-east zone and created an EBS-based AMI in that zone. However, if I try to specify the availability_zone parameter in the configuration, I get the following error:

           

          F, [2011-06-05T10:45:45.642256 #19926] FATAL -- : RuntimeError: You selected eu-west-1b avaibility zone, but your instance is running in us-east-1d zone. Please change avaibility zone in plugin configuration file to us-east-1d (see http://boxgrinder.org/tutorials/boxgrinder-build-plugins/#EBS_Delivery_Plugin) or use another instance in eu-west-1b zone to create your EBS AMI.

           

          I'm not sure I understand the use of this parameter if it can't be different than the availability zone the instance is running in.

           

          There are no meta appliances provided for other zones either. Is there an easy way to create an EBS-backed AMI in another zone that us-east?

          Yes, you understood it correctly. Currently we do not provide meta appliances for different regions. Please read this thread which should explain it in details.

          S3 delivery plugin

           

          Since I've already converted instance-store AMIs to EBS-backed AMIs, I also tried to deliver the appliance with the S3 plugin. However, I always get this error:

           

          I, [2011-06-05T10:56:14.471182 #20044]  INFO -- : Image converted to EC2 format.

          T, [2011-06-05T10:56:14.471416 #20044] TRACE -- : Moving 'build/appliances/x86_64/fedora/15/my_server-f15/ec2-plugin/tmp/my_server-f15.ec2' deliverable to target destination 'build/appliances/x86_64/fedora/15/my_server-f15/ec2-plugin/my_server-f15.ec2'...

          D, [2011-06-05T10:56:14.472618 #20044] DEBUG -- : Platform plugin executed.

          T, [2011-06-05T10:56:14.474719 #20044] TRACE -- : Checking if 'my_server-f15/fedora/15/2.0/x86_64/my_server-f15.ec2.manifest.xml' path exists in my_server...

          T, [2011-06-05T10:56:14.474926 #20044] TRACE -- : Path doesn't exist!

          F, [2011-06-05T10:56:14.475447 #20044] FATAL -- : Errno::ENOENT: No such file or directory - build/appliances/x86_64/fedora/15/my_server-f15/s3-plugin/ami/my_server-f15.ec2.manifest.xml

           

          The problem seemed similar to this post (http://community.jboss.org/message/558438), but trying the proposed solution did not change anything. I have ensured that the AMI tools are installed and working properly, but the error remains.

          Are you sure the file exists on local disk? Have you interrupted the build at some point maybe? Maybe try to run BoxGrinder with -f switch which will remove the previous artifacts and re-execute the build for your appliance definition.

           

          In the meantime please tell me what's the output of rpm -q rubygem-aws.

           

          Thanks!

           

          --Marek

          • 2. Re: Cannot deliver Fedora 15 appliance to EC2 with EBS or S3 plugins
            unknow

            Hello, thanks for the quick answer.

             

            I certainly didn't interrupt the build. The file doesn't appear to exist on the local disk, but that's a file which should be generated by boxgrinder, right?

             

            The output of rpm -q rubygem-aws is:

            rubygem-aws-2.4.2-2.fc15.noarch

             

            I re-started the whole build from scratch, and I noticed that the first time I run it, I get this error:

             

            I, [2011-06-06T07:16:36.952514 #1308]  INFO -- : Image converted to EC2 format.

            T, [2011-06-06T07:16:36.952723 #1308] TRACE -- : Moving 'build/appliances/x86_64/fedora/15/my_server-f15/ec2-plugin/tmp/my_server-f15.ec2' deliverable to target destination 'build/appliances/x86_64/fedora/15/my_server-f15/ec2-plugin/my_server-f15.ec2'...

            D, [2011-06-06T07:16:36.953868 #1308] DEBUG -- : Platform plugin executed.

            T, [2011-06-06T07:16:36.956502 #1308] TRACE -- : Checking if 'my_server-f15/fedora/15/1.0/x86_64/my_server-f15.ec2.manifest.xml' path exists in my_server...

            T, [2011-06-06T07:16:36.956723 #1308] TRACE -- : Path doesn't exist!

            I, [2011-06-06T07:16:36.956982 #1308]  INFO -- : Bundling AMI...

            F, [2011-06-06T07:16:36.957615 #1308] FATAL -- : NoMethodError: undefined method `[]' for nil:NilClass

            /usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.9.1/lib/boxgrinder-build/plugins/delivery/s3/s3-plugin.rb:165:in `bundle_image'

            /usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.9.1/lib/boxgrinder-build/plugins/delivery/s3/s3-plugin.rb:103:in `execute'

            /usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.9.1/lib/boxgrinder-build/plugins/base-plugin.rb:132:in `run'

            /usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.9.1/lib/boxgrinder-build/appliance.rb:144:in `execute_delivery_plugin'

            /usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.9.1/lib/boxgrinder-build/appliance.rb:79:in `execute_plugin_chain'

            /usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.9.1/lib/boxgrinder-build/appliance.rb:90:in `create'

            /usr/lib/ruby/gems/1.8/gems/boxgrinder-build-0.9.1/bin/boxgrinder-build:177

             

            Then, if I try again (without the -f flag), I get the error in my previous post.

             

            I'm going to try again with the -f flag but I don't expect the result will be different than when I ran it from scratch.

             

            EDIT: by the way, I tried this in a local virtual machine with F15 desktop and boxgrinder-build installed manually, and also in the provided meta appliance running on EC2. The error is the same.

            • 3. Re: Cannot deliver Fedora 15 appliance to EC2 with EBS or S3 plugins
              unknow

              I ran it with the -f flag and it still produces the exact same error.

              • 4. Re: Cannot deliver Fedora 15 appliance to EC2 with EBS or S3 plugins
                goldmann

                Please update the system: this will pull new aws gem and BoxGrinder Build 0.9.2. Then re-run the build.

                 

                --Marek

                • 5. Re: Cannot deliver Fedora 15 appliance to EC2 with EBS or S3 plugins
                  unknow

                  I did an update but then boxgrinder-build wouldn't work:

                   

                  /usr/lib/ruby/site_ruby/1.8/rubygems/dependency.rb:247:in `to_specs': Could not find boxgrinder-build (>= 0) amongst [] (Gem::LoadError)

                            from /usr/lib/ruby/site_ruby/1.8/rubygems/dependency.rb:256:in `to_spec'

                            from /usr/lib/ruby/site_ruby/1.8/rubygems.rb:1182:in `gem'

                            from /usr/bin/boxgrinder-build:18

                   

                  It appears it has deleted all my gems. I assume this is a totally unrelated issue as it's happened to be before. (Do you happen to know anything about this problem?)

                   

                  I started from scratch by instantiating a new boxgrinder meta appliance and doing yum update twice until nothing was left to update. I however encountered the exact same error. I think I found the problem in my configuration though. I looked at the code, and when instantiating Aws::S3, this option is passed:

                   

                  :server => REGION_OPTIONS[@plugin_config['region']][:endpoint]

                   

                  However, I used eu-west-1b as the region, which isn't in the REGION_OPTIONS hash. Changing it to eu-west-1 removed the error.

                   

                  I then encountered this error:

                   

                  I, [2011-06-06T18:19:02.418811 #20541]  INFO -- : Opening new HTTPS connection to my_bucket.s3.amazonaws.com:443

                  warning: peer certificate won't be verified in this SSL session

                  W, [2011-06-06T18:19:02.990929 #20541]  WARN -- : ##### Aws::S3Interface returned an error: 403 Forbidden

                  <?xml version="1.0" encoding="UTF-8"?>

                  <Error><Code>RequestTimeTooSkewed</Code><Message>The difference between the request time and the current time is too large.</Message><MaxAllowedSkewMilliseconds>900000</MaxAllowedSkewMilliseconds><RequestId>0431C062C4CF71F0</RequestId><HostId>X43dRnS5zBCy6kyQnn8alY11AIjwHcndE4DjwnVwX5vtHyRhWa1jWv/x4VNGdKIC</HostId><RequestTime>Mon, 06 Jun 2011 22:19:02 GMT</RequestTime><ServerTime>2011-06-06T18:19:04Z</ServerTime></Error> #####

                  W, [2011-06-06T18:19:02.991338 #20541]  WARN -- : ##### Aws::S3Interface request: my_bucket.s3.amazonaws.com:443/ ####

                  F, [2011-06-06T18:19:02.995012 #20541] FATAL -- : Aws::AwsError: RequestTimeTooSkewed: The difference between the request time and the current time is too large.

                   

                  To avoid it, I had to set the server time to the correct UTC time with:

                  cd /etc

                  ln -sf /usr/share/zoneinfo/UTC localtime

                  date -s "$current_time"

                   

                  Now, I encounter this error:

                   

                  I, [2011-06-06T20:05:45.223501 #14738]  INFO -- : Uploading jeos-f15 AMI to bucket 'my_bucket'...

                  D, [2011-06-06T20:05:45.223909 #14738] DEBUG -- : Executing command: 'euca-upload-bundle -U http://s3.amazonaws.com -b my_bucket/jeos-f15/fedora/15/1.0/x86_64 -m build/appliances/x86_64/fedora/15/jeos-f15/s3-plugin/ami/jeos-f15.ec2.manifest.xml -a <REDACTED> -s <REDACTED>'

                  D, [2011-06-06T20:05:45.581861 #14738] DEBUG -- : Checking bucket: my_bucket/jeos-f15/fedora/15/1.0/x86_64

                  D, [2011-06-06T20:05:45.594695 #14738] DEBUG -- : S3ResponseError: 301 Moved Permanently

                  D, [2011-06-06T20:05:45.595117 #14738] DEBUG -- : <?xml version="1.0" encoding="UTF-8"?>

                  D, [2011-06-06T20:05:45.595712 #14738] DEBUG -- : <Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><RequestId>YYYY</RequestId><Bucket>my_bucket</Bucket><HostId>7E2ojNkWuD6O04+deIs+4gAy+KflN9bRdb5NfqhaBh8QzQtHs1rmSZv0eRu1Ika9</HostId><Endpoint>my_bucket.s3.amazonaws.com</Endpoint></Error>

                  F, [2011-06-06T20:05:45.854272 #14738] FATAL -- : AWS::InvalidManifest: HTTP 301 (Moved Permanently) response for URL http://s3.amazonaws.com:80/my_bucket/jeos-f15/fedora/15/1.0/x86_64/jeos-f15.ec2.manifest.xml: check your manifest path is correct and in the correct region.

                   

                  I haven't yet found how to solve this one.

                   

                  Maybe this is relevant: http://www.sitefinity.com/devnet/forums/sitefinity-3-x/set-up-installation/amazon-s3-must-be-addressed-using-the-specified-endpoint.aspx

                  • 6. Re: Cannot deliver Fedora 15 appliance to EC2 with EBS or S3 plugins
                    unknow

                    The above link was indeed relevant.

                     

                    Please forgive me for trying this, but I was in a hurry. I tried the following modification.

                     

                    line 32 of boxgrinder-build-0.9.2/lib/boxgrinder-build/plugins/delivery/s3/s3-plugin.rb:

                    REGION_OPTIONS = {

                            'eu-west-1' => {

                                    :endpoint => 's3-eu-west-1.amazonaws.com',

                                #:endpoint => 's3.amazonaws.com',

                                :location => 'EU',

                                :kernel => {

                                    'i386' => {:aki => 'aki-4deec439'},

                                    'x86_64' => {:aki => 'aki-4feec43b'}

                                }

                            },

                     

                    Then the command worked like a charm. I did not manage to obtain this correct URL with the region and host parameters in the config file. However, I assume there is a way to do it?

                    • 7. Re: Cannot deliver Fedora 15 appliance to EC2 with EBS or S3 plugins
                      goldmann

                      Indeed, this is a bug, could you please file a new ticket? Does it mean that after changing this - you got it working as expected?

                       

                      --Marek

                      1 of 1 people found this helpful
                      • 8. Re: Cannot deliver Fedora 15 appliance to EC2 with EBS or S3 plugins
                        unknow

                        Yes, after changing this, it successfully uploaded and registered the AMI. I could instantiate it and connect to it (I did not do any further tests at this time.)

                         

                        I created a ticket, hopefully with enough information: https://issues.jboss.org/browse/BGBUILD-231

                         

                        By the way, I tried to convert the resulting instance-store image to an EBS-backed image (http://stackoverflow.com/questions/2082724/amazon-ec2-swap-root-instance-store-device-with-ebs-device), but I couldn't seem to reach it through SSH. Do you happen to have any idea why that would be the case? Although I assume it's an issue totally unrelated to BoxGrinder, so I guess this isn't the right forum to ask.

                         

                        Thanks for your help.

                        • 9. Re: Cannot deliver Fedora 15 appliance to EC2 with EBS or S3 plugins
                          goldmann

                          OK, thanks for info!

                           

                          Please look at the console log provided by AWS - you should get more details why it is failing. We can try to debug it here.

                           

                          --Marek

                          • 10. Re: Cannot deliver Fedora 15 appliance to EC2 with EBS or S3 plugins
                            unknow

                            Well, I looked at the console log, but I don't know enough to tell if there's a problem. I've attached it to this post.

                             

                            To convert the boxgrinder-built instance-store image to an EBS-backed image, I tried two procedures:

                             

                            In both cases, I obtained an EBS-backed AMI which starts (according to the EC2 web console) but cannot be reached through SSH. Here are the details of the first procedure the way I tried it. Note that I do not understand each step of the procedure so I simply ran the commands indicated; if there's a mistake in the procedure, I wouldn't know.

                             

                            • build and register a Fedora 15 instance-store AMI with the S3 delivery plugin (my previous problem, which I fixed);
                            • deploy an x86_64 m1.large instance in the eu-west-1b availability zone with that AMI; I do not know which Kernel ID or RAM Disk ID to choose, so I left Default for both fields; the resulting Kernel ID is aki-4feec43b, no RAM Disk ID was indicated;
                            • the instance started and I can connect to it through SSH with the correct key; chkconfig correctly indicates sshd as on;
                              • sshd           0:off 1:off 2:on 3:on 4:on 5:on 6:off
                            • uname -a shows the correct (I assume) OS:
                              • Linux ip-10-226-230-207 2.6.38.6-27.fc15.x86_64 #1 SMP Sun May 15 17:23:28 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

                             

                            Then I create an EBS volume to copy the contents of the ephemeral storage:

                             

                            • create a 10GB EBS volume;
                            • attach the EBS volume to the instance on /dev/sdf (appears as /dev/xvdf);
                            • copy the ephemeral root volume to the EBS volume:
                              • dd bs=65536 if=/dev/xvda1 of=/dev/xvdf
                              • output: 10737418240 bytes (11 GB) copied, 243.093 s, 44.2 MB/s
                            • check the EBS volume for consistency:
                              • fsck /dev/xvdf
                              • output: 79d3d2d4: clean, 17986/655360 files, 212128/2621440 blocks
                            • mount the EBS volume onto the instance:
                              • mkdir /root/ebs-vol && mount /dev/xvdf /root/ebs-vol
                            • remove the /mnt entry from the fstab in the EBS volume:
                              • vi /root/ebs-vol/etc/fstab
                              • I removed this line: /dev/xvdb   /mnt      ext3    defaults         0 0
                            • unmount the EBS volume:
                              • umount /dev/xvdf

                             

                            Now I can register the AMI:

                             

                            • create a snapshot of the EBS volume;
                            • register a new AMI based on the EBS snapshot (with the same EC2 key pair, although it shouldn't matter):
                              • ec2-register -s snap-YYYY -a x86_64 -d "JEOS Fedora 15 (EBS)" -n "JEOS Fedora 15 (EBS)" --kernel aki-4feec43b --region eu-west-1

                             

                            I then tried to launch a t1.micro instance (this is the kind I need) using that AMI. The instance is started successfully (according to the EC2 web console), but I cannot connect to it through SSH.

                            ssh: connect to host ec2-YYY-YYY-YYY-YYY.eu-west-1.compute.amazonaws.com port 22: Operation timed

                             

                            I also tried to connect through an EC2 elastic IP address, same result. These are the last lines of the console log (see attached file for full log):

                             

                            Starting Initialize storage subsystems (RAID, LVM, etc.)...

                            Started Initialize storage subsystems (RAID, LVM, etc.).

                            Starting Initialize storage subsystems (RAID, LVM, etc.)...

                            Started Initialize storage subsystems (RAID, LVM, etc.).

                            Started Show Plymouth Boot Screen.

                            [   96.107078] systemd[1]: Job dev-xvdc.device/start timed out.

                            Starting /data [1;31maborted[0m because a dependency failed.

                            Starting Relabel all filesystems, if necessary [1;31maborted[0m because a dependency failed.

                            [   96.107383] systemd[1]: Job fedora-autorelabel.service/start failed with result 'dependency'.

                            Starting Mark the need to relabel after reboot [1;31maborted[0m because a dependency failed.

                            [   96.107482] systemd[1]: Job fedora-autorelabel-mark.service/start failed with result 'dependency'.

                            [   96.107500] systemd[1]: Job local-fs.target/start failed with result 'dependency'.

                            [   96.107515] systemd[1]: Triggering OnFailure= dependencies of local-fs.target.

                            [   96.108507] systemd[1]: Job data.mount/start failed with result 'dependency'.

                            [   96.108525] systemd[1]: Job dev-xvdc.device/start failed with result 'timeout'.

                            Welcome to emergency mode. Use "systemctl default" or ^D to activate default mode.

                            [   96.275451] systemd[1]: Startup finished in 272ms 200us (kernel) + 4s 708ms 697us (initrd) + 1min 31s 292ms 50us (userspace) = 1min 36s 272ms 947us.

                            Give root password for maintenance

                            (or type Control-D to continue):

                             

                            I would be grateful for any help or pointers.

                             

                            -- Simon

                            • 11. Re: Cannot deliver Fedora 15 appliance to EC2 with EBS or S3 plugins (eu-west-1 availability zone)
                              msavy

                              Your original issue has been resolved in release 0.9.3, please update!

                               

                              http://boxgrinder.org/blog/2011/06/28/boxgrinder-0-9-3-released/