Terraform and AWS Spot Instances

Last update: January 19, 2020

Terraform is my tool of choice for cloud provisioning. The difference compared to the similar tools for cloud provisioning is that Terraform saves the state. Terraform supports AWS and can provision spot instances also. Spot instances are great to save some money in the cloud. They could be used for any service, but it is really hard to manage stateful services like databases. If you have a stateless service spot instances are a great choice. You need to automate spot instance provisioning, otherwise, your service might be down when they get terminated by AWS. You can't control when the instance will be terminated, but you can keep them 'always up' with Terraform and a simple cron job.

Create a Spot Instance

Creating a spot instance with Terraform is almost the same like managing on-demand instances. There are a couple of new arguments and everything else is the same. In this example, I will use aws_spot_instance_request resource instead of aws_instance. I added those three additional arguments:

spot_price = "${var.spot_price}"
wait_for_fulfillment = true
spot_type = "one-time"

To check a full list of available arguments take a look at spot_instance_request docs.

Here is the example of Terraform state for spot instance provisioning (variables are in a separate file):

provider "aws" {
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  region = "${var.region}"
}

resource "aws_spot_instance_request" "web" {
  availability_zone = "${var.availability_zone}"
  ami = "${var.ami}"
  instance_type = "${var.instance_type}"
  user_data = "${file("user-data.txt")}"

  spot_price = "${var.spot_price}"
  wait_for_fulfillment = true
  spot_type = "one-time"

  root_block_device {
    volume_size = "${var.root_ebs_size}"
  }

  connection {
    user = "ubuntu"
    private_key = "${file("/Users/alen/.ssh/id_rsa")}"
    host = "${aws_spot_instance_request.web.public_ip}"
  }

  // Tag will not be added. Below script will copy tags from spot request to the instance using AWS CLI.
  // https://github.com/terraform-providers/terraform-provider-aws/issues/32
  tags {
    Name = "web"
    Env = "dev"
    InstanceType = "spot"
  }

  provisioner "file" {
    source = "set_tags.sh"
    destination = "/home/ubuntu/set_tags.sh"
  }

  provisioner "remote-exec" {
    inline = [
      "bash /home/ubuntu/set_tags.sh ${var.access_key} ${var.secret_key} ${var.region} ${aws_spot_instance_request.web.id} ${aws_spot_instance_request.web.spot_instance_id}"
    ]
  }
}

I had a hard time to set instance tags. Specified tags were added to spot instance request, and not to the instance itself. The workaround is to use a simple bash script with remote-exec provisioner. We can get those tags from spot instance request with AWS CLI and apply them to the instance:

#!/bin/bash

# Install additional requirements
sudo apt-get update && sudo apt-get install -y python-pip
sudo pip install awscli

# Get spot instance request tags to tags.json file
AWS_ACCESS_KEY_ID=$1 AWS_SECRET_ACCESS_KEY=$2 aws --region $3 ec2 describe-spot-instance-requests --spot-instance-request-ids $4 --query 'SpotInstanceRequests[0].Tags' > tags.json

# Set instance tags from tags.json file
AWS_ACCESS_KEY_ID=$1 AWS_SECRET_ACCESS_KEY=$2 aws --region $3 ec2 create-tags --resources $5 --tags file://tags.json && rm -rf tags.json

Above script will be executed with Terraform remote_exec provisioner. Apply this state and your instance is ready:

⚡ terraform apply

Keep the Spot Instance Always Up

You have your instance running, but now we need to check if the instance is up, and if not to run terraform apply again. This can be automated with a simple cron job that will do this check every 5 minutes:

*/5 * * * * cd /home/ubuntu/terraform_files; terraform plan -detailed-exitcode > /dev/null || terraform apply

I hope you found this blog post useful. Please share and subscribe. Stay tuned for the next one.