Cloud-init in Vagrant with Ubuntu 12.10, 13.04

Flyclops’s entire stack runs on AWS, and uses CloudFormation and AutoScale Groups to launch servers on the fly, with no intervention on our part. We don’t use custom AMIs, but vanilla Ubuntu server images (12.10 at the time of writing this) and Ubuntu’s very cool CloudInit functionality to bootstrap building servers with shell scripts. I want to use the same identical method to bootstrap my Vagrant boxes for local development. Here’s how to do it.

I’ll go further in to how we bootstrap – templated shell files that compile in to multipart mime messages, gzipped and put on S3, and a single bootstrap-to-the-bootstrap script being passed in to the server as user data – in another post. But I wanted my local servers to run almost identical boot scripts via Vagrant so my dev environment was almost identical to my production environment.

Ubuntu provides cloud images that are used in EC2, as well as OpenStack, Rackspace, etc., and even have versions for Vagrant. The Vagrant versions have their own cloud-init boot scripts built in to set up networking and whatnot… but I want them to then run my scripts. Obviously.

Vagrant has this really nice thing where you can supply a shell script to run the first time your boxes start. Simply put something similar to this to have it run your file:

  # Run our shell script on provisioning
  config.vm.provision :shell, :path => "vagrant_build.sh"

Next we want to provide a very small shell script that clears out cloud-init run-info and provides us access to have it run again. Here’s the script, we’ll step through it below.

PLEASE NOTE: This method of working with cloud-init does not work on Ubuntu 12.04 or earlier – cloud-init was significantly updated between 12.04 and 12.10.

# Check to see if we have done this already
if [ -f /.vagrant_build_done ]; then
    echo "Found, not running."
    exit
fi

# Make the box think it hasn't init-ed yet
rm -rf /var/lib/cloud/instance/*
rm -rf /var/lib/cloud/seed/nocloud-net/user-data

# Seed our own init scripts
cat << 'END_OF_FILE_CONTENTS' > /var/lib/cloud/seed/nocloud-net/user-data
Content-Type: multipart/mixed; boundary="===============apiserversStackMultipartMessage=="
MIME-Version: 1.0

--===============apiserversStackMultipartMessage==

#include
https://someS3bucket.s3.amazonaws.com/somefolder/vagrant/someDateStampedFile.gz
--===============apiserversStackMultipartMessage==--
END_OF_FILE_CONTENTS

# Re-run cloud-init
cloud-init init
cloud-init modules --mode init
cloud-init modules --mode config
cloud-init modules --mode final

# Do not let this run again
touch /.vagrant_build_done

First things first – we don’t want this to optionally repeat rerunning cloud-init, so we check for a record that this has run before:

# Check to see if we have done this already
if [ -f /.vagrant_build_done ]; then
    echo "Found, not running."
    exit
fi

Assuming we don’t find anything, we’re off to the races. Lines 8-9 clean up the existing cloud-init trail, almost re-virgining the machine. That said, we’re going to re-use the meta-data file from the vanilla image’s nocloud-net seed, and simply supply our own user-data. We also clear out the “instance” directory so our new scripts can be written there by cloud-init.

# Make the box think it hasn't init-ed yet
rm -rf /var/lib/cloud/instance/*
rm -rf /var/lib/cloud/seed/nocloud-net/user-data

Lines 12-23 do the heavy-lifting of moving a script in to place for us. Again, we’re putting this in

/var/lib/cloud/seed/nocloud-net/user-data

and using cat to move the contents of the file for us.

Note: This method of using bash scripts to write files is ugly, but works reliably. I use this method a lot – but I have a custom Jinja2 filter that writes out the ugly for me, keeping my template files nice and clean.

# Seed our own init script
cat << 'END_OF_FILE_CONTENTS' > /var/lib/cloud/seed/nocloud-net/user-data
Content-Type: multipart/mixed; boundary="===============apiserversStackMultipartMessage=="
MIME-Version: 1.0

--===============apiserversStackMultipartMessage==

#include
https://someS3bucket.s3.amazonaws.com/somefolder/vagrant/someDateStampedFile.gz

--===============apiserversStackMultipartMessage==--
END_OF_FILE_CONTENTS

Notice on line 18, we’re actually writing out

#include

which tells cloud-init to “include” scripts at that url. According to the docs,

the content read from the URL can be gzipped, mime-multi-part, or plain text

so I store a gzipped multipart mime file. Works like a charm, and for EC2, gets us around the user-data size limit for supplying scripts to cloud-init.

Now comes the fun – with cloud-init cleaned out and our new user-data in, we simply force-run cloud-init again.

# Re-run cloud-init
cloud-init init
cloud-init modules --mode init
cloud-init modules --mode config
cloud-init modules --mode final

First we initialize it, then run “modules” through initialization, configuration, and finalization. The line

cloud-init modules --mode final

is the most important for us. Through initialization and configuration, cloud-init has downloaded our gzip file from S3, has unzipped and separated out all of the parts of our multipart-mime message in to component shell scripts, but has not yet run those shell scripts. Line 29 asks cloud-init to now run those scripts.

Line 32 is just so that if Vagrant re-executes this build script at any point in time, we don’t run cloud-init through the initialization process again.

# Do not let this run again
touch /.vagrant_build_done

So that’s it. Despite the length of this post, it’s a very short script. I should note that you can also add in any other initialization needed specifically for Vagrant boxes after this – like sym-linking in /vagrant (the shared directory) in to a particular location, etc., to finalize the box for local development.

Let me know what you think in the comments below!

  • nitzer70

    Very cool tips! Thanks a lot

  • Sonia

    Great article! how ever something is missing and not working for me. I did the following:
    1. Cleaned out both the directories as per line 8 and 9.
    2. vi

    /var/lib/cloud/seed/nocloud-net/user-data —> put a sample contents like this :
    #cloud-config
    runcmd:
    – [ cp /opt/test.txt /tmp ]

    3. cloud-init init —> this command brings back the instance/ directory (link) for me and is pointing to the instances/ contents which has the old user-data I provided while VM provisioning.

    4. When run the next 3 modules – init , configure and final –> it starts executing the old user-data file instead of what I added in seed/nocloud-net/user-data.

    I cannot find if these modules ever got the new user-data file which I added …and still runs the old script….any suggestions ?