Terraform Packer Tips

Building some packer images over the last couple of weeks, there were a few little problems I had to solve.

• apt-get fails with the message no installation candidate for a package that should exist
• git over ssh wants you to verify host keys before pulling down a repo
• I want to pull code from multiple private repos as part of the build process, but a github.com deploy key can only be used with a single repo

AWS: wait for Cloud Initialization to complete

I had a packer configuration that started installing packages like this:

provisioner “shell” {
  inline = [
    “sudo apt-get update -yqq”,
    "sudo apt install -yqq aptitude”,
    “sudo aptitude update”,
    “sudo aptitude install -y git ec2-api-tools”
  ]
}

Looks great, but I found that most (but not all) of my packer runs would abort with a message like:

Package aptitude is not available, but is referred to by another package.
E: Package 'aptitude' has no installation candidate

Now, I know these packages are real packages, so what gives?

The short answer is that the EC2 instance hadn’t yet finished initializing by the time packer ran these commands. By adding a pause, we wait until the initialization is done, which eliminates this failure. AWS actually provides a flag file that we can wait for to know that the initialization is complete.

provisioner “shell” {
  inline = [
    "timeout 60s bash -c \"while ! [ -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting on cloud-init...'; sleep 2; done\"",
    “sudo apt-get update -yqq”,
    "sudo apt install -yqq aptitude”,
    “sudo aptitude update”,
    “sudo aptitude install -y git ec2-api-tools”
  ]
}

With this change, the packer build no longer fails with a message about no installation candidate. Yay!

Pre-load github.com SSH fingerprint

In my packer config, I load a bootstrap.sh script onto the instance and then execute it to download some code from github.com . Any time you connect to an ssh host that isn’t listed in ~/.ssh/known_hosts, ssh will ask you if you want to accept their key.

The authenticity of host '10.2.1.29 (10.2.1.29)' can't be established.
ECDSA key fingerprint is SHA256:2AFwdL44LriTQJlPt1L4ea5IWQo001I7iK37JS9EV1c.
Are you sure you want to continue connecting (yes/no)?

I’d like to skip this step at packer build time, but to be secure I don’t want to auto-accept the key.

This is possible by building your ~/.ssh/known_hosts file in advance. Before building your image, fetch the keys:

$ ssh-keyscan github.com > known_hosts

You can then verify the fingerprints against the published ones: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints

Once that’s done, you can load the known_hosts file into ~/.ssh/known_hosts, and you’re done. Running git as part of your packer build will validate the remote’s ssh key against the known_hosts file and help guard against some attacks.

(It also means that if github.com does change their keys, your build will break, but I’d say this is a feature, not a bug.)

Fetch code from multiple git repos during packer build

Your packer build can pull code down from github at build time, but one thing you will run into is the fact that your deploy keys can only be used for a single repo. This only matters for private repos; for a public repo the deploy keys don’t matter.

Once you generate the key pair, and then generate the fingerprint, that fingerprint can only be added to a single repo as a deploy key. Github gives you an error if you try to add it to a second repo.

You’ll need to create two keys, put both of them into your image, and then fetch data from git while specifying which key you want used. It’s easy!

Start out by generating the two keys, and adding each one to one of the private github repos you want to fetch from.

Next, load the keys into your image. In my examples below, the packer build has already created a deploy user which does all the work.

provisioner "file" {
  destination = "/tmp/"
  sources = [
    "./files/keys/key1",
    "./files/keys/key2"
  ]
}
provisioner "shell" {
  inline = [
    // these keys are used to authenticate with github (deploy keys)
    "sudo mv /tmp/key1 /home/deploy/.ssh",
    "sudo mv /tmp/key2 /home/deploy/.ssh",
    "sudo chmod 600 /home/deploy/.ssh/*",
    "sudo chown -R deploy:deploy /home/deploy/.ssh"
    // i have much of my bootstrap in a shell script
    "sudo -H -u deploy bash -l -c 'cd /home/deploy && ./bootstrap.sh'"
  ]
}

Now, in my bootstrap script, we can specify which key should be used with which repo:

#!/bin/bash
cd $HOME
GIT_SSH_COMMAND='ssh -i /home/deploy/.ssh/key1 -o IdentitiesOnly=yes' git clone git@github.com:{account}/{repo}.git

The IdentitiesOnly=yes bit ensures that git uses only the specified key.

Hope these tips help!