Hey,

I just got an idea about a tool, and given that I think it’s a type of thing that pairs nicely with a blog to drive people to it, I decided to create one.

Given that I already have everything set up for this one (ops.tips, I mean), I thought about sharing my setup with you so that you can also adopt it - if you think it’s cool.

ps.: this is not meant to be advertising for AWS. I’m just talking about my use-case here.

update: added a CDN invalidation step to the Travis-CI deployment script - thanks a lot Corey Quinn!

How it looks

The whole process of delivering the content of my blog is based on four steps:

  1. create the content (markdown)
  2. build and publish the content (send generated HTML to somewhere)
  3. serve the build result (accept connections and serve HTML)
  4. invalidating old objects in the edge caches

Naturally, the first case doesn’t matter for this blog post.

Building and publishing the content

All these blog posts are plain markdown files with some custom templating rules.

To get markdown processed and HTML generated I make use of Hugo.

You define a theme for your content and then based on the theme and what you write in some markdown files it generates a specific HTML.

Illustration of the publishing process

The code that does so (automatically in travis-ci after I push to a specific branch) is fairly simple:

# Build the markdown files and output
# (verbosely) information about the 
# process.
hugo -v

# Go to the directory where the static
# files now live
cd ./public 

# Send the new files to the configured S3 
# (and delete the old ones that are not in
# the new desired state) with a special
# max-age header configuration so that both
# browsers and CDNs cache it for some time.
aws s3 sync \
        --delete \
        --cache-control max-age=10800 \
        ./ s3://$BUCKET

# Create a new invalidation that will simply
# invalidate any objects that we have in our
# website bucket such that we serve fresh content
# to the users that are requests files from the
# edge locations
#
# ps.: users that already downloaded our 
#      resources will still have then until
#      the browser cache gets invalidated
aws cloudfront \
        create-invalidation \
        --distribution-id $DISTRIBUTION_ID \
        --paths '/*'

Once the code has been built and sent to S3, it comes the time to post the content to S3.

Serving the website

As the final htmls lands on S3, it’s a matter of serving it.

S3 allows you to serve your content directly from it (see Hosting a Static website on S3). The problem of doing so is that we can’t configure it very much. It’s not flexible at all.

Another drawback from serving directly from it is that if we have users from a different continent trying to get content from our website, they’ll have to make requests that land in our continent, making their response times not very good.

A solution to that (which AWS already provides and that integrates very smoothly with S3) is to use a CDN (in this case, CloudFront).

How caching works for requests

To make clear why it makes sense to use CloudFront (or any other CDN, to be honest), we can think of how browsers interact with our servers.

The browser-server interaction

When the user requests our website (say, myblog.com), it has first to get at least one IP that tells it the location to establish a TCP connection to (so that it can make an HTTP request to request a file).

This process is called name resolution.

Illustration of the name resolution process

Once the browser has the IP of the server to contact, it starts a TCP connection which, in our case, will require negotiating some parameters and exchanging a certificate which the browser decides whether it’s trusty or not.

In the case that the browser trusts the certificate received, it can then start sending application-layer requests (in our case, HTTP2 frames).

Illustration of the client establishing a connection

Having a service like CloudFront makes a huge difference here as this process involves several roundtrips (there’s a great article from CloudFlare about this - Introducing Zero Round Trip Time Resumption (0-RTT)). The less it takes to perform a roundtrip, the better.

Because a CDN has all these points of presence (PoP) all over the world (and it has our certificate in all of them - at least at some point), the TLS termination happens much closer to the client.

Illustration of CloudFront all over the globe

In the case that the PoP also has the file you need, it can serve it directly from there without having to have your file coming from a distant location (S3).

Illustration of the points of presence at the edges serving the content

Now you might ask me, does it matter? In a typical scenario where you just launch a blog post, do you have all those cache hits?

From the CloudFront data I can tell that yeah, it definitely matters:

CloudFront statistics when HackerNews starts hitting the blog

Even though my blog is pretty slim and has not many large images (as I optimize them before publishing), it’s still a decent amount of traffic that is not being directed to a far origin.

The actual thing - terraform module

To put all of this into practice, I make use of Terraform to create the necessary resources in AWS.

The module is made of three files: outputs.tf, inputs.tf and main.tf.

inputs.tf defines the variables that are required by the module:

variable "domain" {
  type        = "string"
  description = "name of the domain for the distribution and to alias route53 to"
}

variable "bucket" {
  type        = "string"
  description = "name of the bucket that will hold the files (access logs goes to {bucket}-log"
}

variable "storage-secret" {
  type        = "string"
  description = "a secret user-agent that is sent to all requests from CF to S3"
}

variable "acm-certificate-arn" {
  type        = "string"
  description = "the aws resource number of the certificate that has been manually provisioned"
}

Those variables are used in the main.tf which manages the resources:

# Creates the bucket that will hold logs about
# access to S3 objects.
# The number of access shouldn't be very big given
# that we're caching things at the edge.
#
# In theory, it should have, per day, a maximum of
# number_of_pops*number_of_objects log lines coming
# from S3.
#
# As we also log all the requests from CloudFlare, we
# should have `n` entries for CDN access (as the requests
# are recorded).
#
# To not keep all of these log records that
# we will almost never look at we transition
# these log files to infrequent access storage
# to reduce costs after 30 days.
resource "aws_s3_bucket" "website_log_bucket" {
  bucket        = "${var.bucket}-logs"
  acl           = "log-delivery-write"
  force_destroy = true

  lifecycle_rule {
    enabled = true

    transition {
      days          = "30"
      storage_class = "STANDARD_IA"
    }
  }
}

# The main bucket that holds the actual HTML files
# that are generated.
#
# It defines what's an error document (404.html) and
# what's the `index.html` of it (if we wanted to access
# the website from S3 directly).
#
# It has versioning enabled such that when we modify
# or delete files they are not really deleted and then
# we can go back if we want.
#
# After 15 days we expire the non-current versions as
# at that point it's probably all good there.
#
# Here we also set the bucket policy so that we only
# allow the gets from those having a secret user-agent
# and is a request from aws.
resource "aws_s3_bucket" "website_bucket" {
  bucket = "${var.bucket}"

  website {
    index_document = "index.html"
    error_document = "404.html"
  }

  versioning {
    enabled = true
  }

  lifecycle_rule {
    enabled = true

    noncurrent_version_expiration {
      days = "15"
    }
  }

  logging {
    target_bucket = "${aws_s3_bucket.website_log_bucket.id}"
    target_prefix = "website-bucket-logs/"
  }

  policy = <<POLICY
{
    "Statement": [
        {
            "Action": [
                "s3:GetObject"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:UserAgent": "${var.storage-secret}"
                }
            },
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Resource": "arn:aws:s3:::${var.bucket}/*",
            "Sid": "PublicReadAccess"
        }
    ],
    "Version": "2012-10-17"
}
POLICY
}

# A user that we can use to perform the deployment.
#
# The idea here is that if we want to put the deployment
# process (i.e., uploading of new contents to S3) in a
# CI system we don't want to give it many privileges -
# just the bare minimum which is `put` and `list` for that
# bucket.
resource "aws_iam_user" "main" {
  name = "${var.bucket}-deployer"
}

# Create ACCESS and SECRET keys that we can feed our tools
resource "aws_iam_access_key" "main" {
  user = "${aws_iam_user.main.name}"
}

# Assign a policy to the user that restricts its actions.
resource "aws_iam_user_policy" "main" {
  name = "${aws_iam_user.main.name}-deploy-policy"
  user = "${aws_iam_user.main.name}"

  policy = <<POLICY
{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::${var.bucket}"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::${var.bucket}/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}
POLICY
}

# Create the CloudFront distribution that:
# - has IPV6 enabled
# - is spread across all the PoPs (PriceClass_All)
# - has a default root object for our `/` requests
# - sends requests to the origin with our secret header
# - is tied to the S3 origin
# - has logging configured
# - has caching enabled
# - enforces https
# - sets a minimum TLS version
resource "aws_cloudfront_distribution" "website_cdn" {
  enabled             = true
  is_ipv6_enabled     = true
  price_class         = "PriceClass_All"
  http_version        = "http2"
  default_root_object = "index.html"

  "origin" {
    origin_id   = "origin-bucket-${aws_s3_bucket.website_bucket.id}"
    domain_name = "${aws_s3_bucket.website_bucket.website_endpoint}"

    custom_origin_config {
      origin_protocol_policy = "http-only"
      http_port              = "80"
      https_port             = "443"
      origin_ssl_protocols   = ["TLSv1"]
    }

    custom_header {
      name  = "User-Agent"
      value = "${var.storage-secret}"
    }
  }

  logging_config {
    include_cookies = false
    bucket          = "${aws_s3_bucket.website_log_bucket.bucket_domain_name}"
    prefix          = "cdn-logs/"
  }

  custom_error_response {
    error_code            = "404"
    error_caching_min_ttl = "360"
    response_code         = "404"
    response_page_path    = "/404.html"
  }

  "default_cache_behavior" {
    min_ttl                = "0"
    default_ttl            = "3600"
    max_ttl                = "3600"
    target_origin_id       = "origin-bucket-${aws_s3_bucket.website_bucket.id}"
    viewer_protocol_policy = "redirect-to-https"
    compress               = true

    allowed_methods = [
      "GET",
      "HEAD",
      "DELETE",
      "OPTIONS",
      "PATCH",
      "POST",
      "PUT",
    ]

    cached_methods = [
      "GET",
      "HEAD",
    ]

    "forwarded_values" {
      query_string = "false"

      cookies {
        forward = "none"
      }
    }
  }

  "restrictions" {
    "geo_restriction" {
      restriction_type = "none"
    }
  }

  "viewer_certificate" {
    acm_certificate_arn      = "${var.acm-certificate-arn}"
    ssl_support_method       = "sni-only"
    minimum_protocol_version = "TLSv1"
  }

  aliases = [
    "${var.domain}",
  ]
}

# Retrieve information about the manually configured
# public hosted zone.
data "aws_route53_zone" "primary" {
  name         = "${var.domain}."
  private_zone = false
}

# Creates an ALIAS record that ties our domain to
# the CloudFlare distribution.
#
# We could also have a resource that CNAMEs `www` such
# that we'd have `www` set for our website as well.
resource "aws_route53_record" "cdn-alias" {
  zone_id = "${data.aws_route53_zone.primary.zone_id}"
  name    = "${var.domain}"
  type    = "A"

  alias {
    name                   = "${aws_cloudfront_distribution.website_cdn.domain_name}"
    zone_id                = "${aws_cloudfront_distribution.website_cdn.hosted_zone_id}"
    evaluate_target_health = false
  }
}

Aside from running a main.tf that imports the module and creates the resource I just have to do two things manually:

  • tell GoDaddy to use Route53 as the nameservers for this domain (such that we can make use of ALIAS records to have the domain tied to CloudFront);
  • accept the terms of service for the certificate generation.

And that’s it! Push to git and everything is live.

Closing thoughts

The primary objection to doing things this way is that there are already great players out there to serve static files, and I get the point. For me, however, being able to guarantee a great performance, control each step, get all the metrics I need and still have the same great user experience, makes publishing with AWS great - set it up once (takes 30m) and then never worry again.

I think now there’s only one thing that I’d improve:

  • automatic cache invalidation when publishing: would allow me to make sure clients are getting the last published content at the time I post (supposing I want to quickly edit something or make sure that the index.html that lists all pages is up to date on all locations).

Update: this is actually pretty simple and doens’t require any Lambda stuff. Corey Quinn from LastWeekInAWS gave me an excelent idea: use the AWS CLI to perform the invalidations. This way we can tie it to the travis-ci deployment script and we’ll have the invalidations on every push. Pretty neat!

To achieve that I could have an AWS lambda function that is triggered when I finish pushing to S3. It hasn’t been an issue so far as I’ve only needed to expire the cache twice (in those cases I just went to the console and clicked few buttons, no big deal).

If you liked to post and would want to see more content related to AWS and coding in general, make sure to subscribe to the mailing list below. You can also reach me at Twitter at any time.

Have a good one!

finis