Hey,

with the upcoming release of HAProxy 1.8 (see the blog post at haproxy.com) it’ll be possible to keep your stack behind the goodness of http2 without changing your code at all.

That’s pretty cool as you can have very perceptive differences under real-life scenarios. One thing to notice is that browsers only establish these connections if you’re HTTPS ready, and that means having TLS certificates in your load-balancer (or regular server).

Here are my 2 cents on how you can have a fully functioning HAProxy set up with certificate generation via Letsencrypt.

Dependencies

There are a handful of dependencies that need to be in place: certbot, lua, openssl-dev,zlib-dev, libpcre-dev and haproxy itself. If you already have all of those set, just skip to the next session: HAProxy Configuration section.

Here I’m making use of Ubuntu zesty (17.04) so there might be some differences between what I document here and your OS specifics.

Certbot

Start by adding certbot’s apt repository:

# after issuing the command press `enter` to accept
sudo add-apt-repository ppa:certbot/certbot

 This is the PPA for packages prepared by Debian Let's Encrypt Team and backported for Ubuntu(s).
 More info: https://launchpad.net/~certbot/+archive/ubuntu/certbot
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keybox '/tmp/tmp028qrfr4/pubring.gpg' created
gpg: /tmp/tmp028qrfr4/trustdb.gpg: trustdb created
gpg: key 8C47BE8E75BCA694: public key "Launchpad PPA for certbot" imported
gpg: Total number processed: 1
gpg:               imported: 1
OK

then update:

sudo apt update -y

Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu zesty InRelease
Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu zesty-updates InRelease [89.2 kB]  
Get:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu zesty-backports InRelease [89.2 kB]
Get:4 http://ppa.launchpad.net/certbot/certbot/ubuntu zesty InRelease [21.3 kB]         
Hit:5 http://security.ubuntu.com/ubuntu zesty-security InRelease        
...

and now install the desired package, python-certbot:

sudo apt install -y python-certbot

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  certbot dialog python-acme python-asn1crypto python-certifi python-chardet
  python-configargparse python-configobj python-cryptography python-dialog
...


certbot help

-------------------------------------------------------------

  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
it will attempt to use a webserver both for obtaining and installing the
certificate. The most common SUBCOMMANDS and flags are:

...

Once we have that we know that certbot is ready to request certificates from Letsencrypt for us.

HAProxy

Before installing HAProxy itself we need to get its dependencies right. HAProxy by itself can’t serve content from a directory like a static hosting web server would so to do that we must make use of lua which allows us to very easily extend HAProxy functionality. Then, the first dependency we’ll get is lua.

# set the version of Lua that we want to install.
# is a variable that we can reference later.
LUA_VERSION=5.3.3

# install a development library that lua depends on
sudo apt install -y \
        libreadline-dev

# fetch lua's source code for the version we want
curl \
        -SOL https://www.lua.org/ftp/lua-$LUA_VERSION.tar.gz

# create a directory to hold that source code
sudo mkdir \
        -p /usr/src/lua 

# extract the source code to the directory we want
sudo tar \
        -xzf lua-$LUA_VERSION.tar.gz \
        -C /usr/src/lua --strip-components=1 

# build Lua using a concurrency equivalent to the number
# of processors we have (and targetting Linux)
sudo make \
        -C /usr/src/lua \
        -j "$(getconf _NPROCESSORS_ONLN)" \
        linux 

# install it so that it's widely accessible
sudo make \
        -C /usr/src/lua \
        install

To proceed with the HAProxy installation we need to know where the lua headers and compiled library are:

# search for the headers
find /usr/local/include -name "*lua*"
/usr/local/include/lua.h
/usr/local/include/lualib.h
/usr/local/include/lua.hpp
/usr/local/include/luaconf.h

# search for the lib
find /usr/local/lib -name "*lua*"
/usr/local/lib/liblua.a
/usr/local/lib/lua

The next dependencies can all be fetched from apt though. We need three:

  • OpenSSL: for full TLS support (e.g, SNI extension);
  • PCRE: to not rely on libc’s regex support but the more common Perl regex
  • ZLIB: for providing deflate and gzip compression algorithms
sudo apt install -y \
        libpcre3-dev \
        libssl-dev \
        zlib1g-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
...

Now that our dependencies are all ready to be used we can proceed with HAProxy itself.

For obtaining the source code we can head to the downloads page (haproxy.org/download/1.8/src/) and fetch the latest version (I’m getting the last release candidate of 1.8 because this version gives us HTTP2 support, which is something I plan to write about soon).

# fetch the .tar.gz of the source code of haproxy 
curl -SOL http://www.haproxy.org/download/1.8/src/haproxy-1.8-rc3.tar.gz

# decompress it to the current directory
tar xzf ./haproxy-1.8-rc3.tar.gz

Once everything is in place it’s just a matter of compiling it with some flags that signalize the build process what is our environment and where it can find the Lua dependency:

# get into the source code directory
cd ./haproxy-1.8-rc3/

# build the code with Lua, gzip compression, PCRE regex
# targetting a recent kernel.
make \
        TARGET=linux2628 \
        USE_OPENSSL=1 \
        USE_ZLIB=1 \
        USE_PCRE=1 \
        USE_LUA=1 \
        LUA_LIB_NAME=lua \
        LUA_LIB=/usr/local/lib/ \
        LUA_INC=/usr/local/include

# make it available from `$PATH` so that
# we can call `haproxy` from anywhere and
# also have `man` pages set
sudo make install

If you wonder why TARGET=linux2628, head to the Makefile at haproxy’s source and check this:

ifeq ($(TARGET),linux2628)
  # This is for standard Linux >= 2.6.28 with 
  # netfilter, epoll, tproxy and splice.
  USE_NETFILTER         = implicit
  USE_POLL              = implicit
  USE_EPOLL             = implicit
  USE_TPROXY            = implicit
  USE_LIBCRYPT          = implicit
  USE_LINUX_SPLICE      = implicit
  USE_LINUX_TPROXY      = implicit
  USE_ACCEPT4           = implicit
  USE_FUTEX             = implicit
  USE_CPU_AFFINITY      = implicit
  ASSUME_SPLICE_WORKS   = implicit
  USE_DL                = implicit
  USE_THREAD            = implicit
else

In summary, it means that it’ll use some recent kernel capabilities to improve our performance.

After the compilation finishes you should have something like this:

./haproxy -vvvv
HA-Proxy version 1.8-rc3-34650d5 2017/11/11
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-null-dereference -Wno-unused-label
  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Encrypted password support via crypt(3): yes
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with multi-threading support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
	[SPOE] spoe
	[COMP] compression
	[TRACE] trace

From the output, we can make sure that what we wanted has been properly compiled. Now we can move to the HAProxy configuration.

HAProxy Configuration

With all the dependencies installed we can proceed with the actual HAProxy configuration.

Because we need to have HAProxy performing something that it can’t do by default as mentioned - serve static files from a directory - we need to add a Lua plugin that does the job. That’s needed to serve the challenge that letsencrypt gives us when checking if we can serve content from a given domain (webroot).

Thanks to Jan (github.com/janeczku), we don’t have to write our own:

# get out of the haproxy directory
cd ../

# fetch the lua script code that will serve the challenges
# placed by certbot at a specific directory (webroot)
git clone https://github.com/janeczku/haproxy-acme-validation-plugin

# go to the repository directory and check what are the 
# files that we have there
cd ./haproxy-acme-validation-plugin
tree
.
├── LICENSE
├── README.md
├── acme-http01-webroot.lua
├── cert-renewal-haproxy.sh
└── haproxy.cfg.example

In the acme-http01-webroot.lua make sure you set the non_chroot_webroot variable to a location where we’ll set certbot to put challenges on:

--
-- Configuration
--
-- When HAProxy is *not* configured with the 'chroot' option you must set an absolute path here and pass 
-- that as 'webroot-path' to the letsencrypt client

acme.conf = {
	["non_chroot_webroot"] = "/tmp/webroot"
}

With the Lua plugin in place all we need to do next is specify that path to the plugin in our haproxy.cfg:

# notice that we're specifying in 
# `lua-load` the location of the Lua
# script. Make sure you reference it
# according to your configuration.
global
                maxconn                   8192
                log                       127.0.0.1  local0
                tune.maxrewrite           16384
                tune.bufsize              32768
                tune.ssl.default-dh-param 2048
                max-spread-checks         200
                spread-checks             5
		lua-load                  /home/ubuntu/haproxy-acme-validation-plugin/acme-http01-webroot.lua


# Configure some default values that
# frontends and backends can inherit
# Here I'm setting some dummy timeouts.
# In another blog post we can go through
# some real values there.
defaults
                log       global
                retries   3
                option    redispatch
                option    dontlog-normal
                mode      http

                timeout   http-request 10m
                timeout   client 10m
                timeout   connect 10m
                timeout   server 10m
                timeout   http-keep-alive 10m
                timeout   tunnel 10m
                timeout   client-fin 10m
                timeout   server-fin 10m

# The HTTP frontend serving only as a way of redirecting traffic
# to port :443 where it can serve the real content via HTTPS
# and also accept the requests from letsencrypt.
# The big thing here is `use-service` which essentially means that
# when the `url_acme_http01` ACL is `true` it'll execute the lua 
# script that we registered.
# The name `lua.acme-http01` comes from the lua script itself:
#
#       core.register_service("acme-http01", "http", acme.http01)
#
frontend        http
		bind		*:80
                acl		url_acme_http01 path_beg /.well-known/acme-challenge/
                http-request	use-service lua.acme-http01 if METH_GET url_acme_http01
                redirect	scheme https if METH_GET !url_acme_http01

# Serve the HTTPS traffic.
# For each request that comes it takes the server name indicated 
# in SNI (the TLS extension that gives to an encrypted connection the 
# equivalent of a Host header) it looks on the list of certificates
# that it loaded from the `crt-list` file and then uses that certificate.
# As we're offloading the TLS resolution from the backend to the 
# frontend, `backend_test` will receive unencrypted content as
# if there was a plain TCP connection coming (no need to deal with
# certificates there - only HAProxy has to care about it).
frontend        https
		bind		*:443
                default_backend backend_test


# Dummy backend connecting haproxy to a server on the same machine.
# The server at `localhost:8080` is a simple HTTP server expecting
# regular non-encrypted connections.
backend         backend_test
                server test-server localhost:8080

To make sure we got our configuration right we can make use of the -c flag of HAProxy to perform some sort of “dry run” and only check the config. Assuming we have the configuration at ~/haproxy.cfg:

haproxy --help 
...
Usage : haproxy -f <cfgfile|cfgdir>] [opt]
        ...
        -c check mode : only check config files and exit
        ...

haproxy -c -f ~/haproxy.cfg
Configuration file is valid

As the configuration is valid we can start HAProxy with a privileged user (as it’s going to bind on ports 80 and 443):

sudo haproxy -f ./haproxy.cfg
[info] 328/123916 (374961) : [acme] http-01 plugin v0.1.1

Now that we got HAProxy working we can check if the lua plugin is really receiving the requests when we hit port 80 on the expected path:

# in a separate terminal, make a request to the
# haproxy instance on the path which should respond
# to letsencrypt requests
curl localhost:80/.well-known/acme-challenge/aa
resource not found

# on the HAProxy terminal, check the logs
[warning] 328/124202 (374961) : [acme] http-01 token not found: aa (client-ip: 127.0.0.1)

As it’s all working as expected we can request a certificate to letsencrypt.

# Create the directory where challenges will
# be placed. This is the directory that you have
# to be set earlier in the lua script in the
# `acme.conf` field.
mkdir /tmp/webroot

# make certbot initiate the certificate generation
# process using the webroot method, placing challenges
# at the directory /tmp/webroot and creating the certificate
# for the domain `cirocosta.com`
sudo certbot certonly --webroot -w /tmp/webroot -d cirocosta.com

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
...
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/cirocosta.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/cirocosta.com/privkey.pem
   Your cert will expire on 2018-02-23. To obtain a new or tweaked
...

With the certificates in hand now we have to create a new file that HAProxy can use to terminate the TLS connections. This new file is a concatenation of the private key and the certificate we received from letsencrypt:

# concatenate those files (as I used the root user to
# provision them - via certbot - they were created as
# root, so I must be `root` to see them).
sudo cat \
        /etc/letsencrypt/live/cirocosta.com/privkey.pem \
        /etc/letsencrypt/live/cirocosta.com/fullchain.pem > \
                ~/cirocosta.com

Then update the haproxy.cfg file to make use of the certificate when binding to port 443:

 frontend        https
#                bind		*:443   (before)
                 bind		*:443   ssl crt /home/ubuntu/cirocosta.com
                 default_backend backend_test

Then perform a soft-reload HAProxy after checking if the configuration is fine (it checks the cert):

# check the configuration
haproxy -c -f ~/haproxy.cfg 
Configuration file is valid

# soft-reload the current instance by creating a new
# one passing the pid of the old instance via the `-sf`
# flag
sudo haproxy \
        -f ./haproxy.cfg \
        -sf $(pidof haproxy)
[info] 328/131241 (376196) : [acme] http-01 plugin v0.1.1

# in the terminal where the old HAProxy was running
# you should see the following (perfectly fine) messages:
[WARNING] 328/131241 (374961) : Stopping frontend http in 0 ms.
[WARNING] 328/131241 (374961) : Stopping frontend https in 0 ms.
[WARNING] 328/131241 (374961) : Stopping backend backend_test in 0 ms.
[WARNING] 328/131241 (374961) : Proxy http stopped (FE: 3 conns, BE: 0 conns).
[WARNING] 328/131241 (374961) : Proxy https stopped (FE: 0 conns, BE: 0 conns).
[WARNING] 328/131241 (374961) : Proxy backend_test stopped (FE: 0 conns, BE: 0 conns).

And that’s it!

Even without a properly configured server, we should already be able to see that the HTTPS setup is fine:

Image of Google Chrome with HTTPS working

Closing thoughts

Even though the guide is quite extensive if you consider the process of building HAProxy and Lua from scratch, there’s not much going on:

  1. Make HAProxy serve the challenges at port 80 on a specific path that letsencrypt expects
  2. Concatenate certificates and private keys
  3. Update the HAProxy configuration
  4. Reload HAProxy

Once that’s automated you’re ready to serve your customers with HTTPS / TLS without dropping connections and having to pay for certificates.

Even if the part of setting up the dependencies seem like too much you can tailor a Dockerfile that does all of that and you’re good to go.

If you enjoyed the guide and are willing to learn more about related content, make sure you subscribe to the newsletter. In case you find mistakes I made or think there’s something to improve, just let me know, I’m cirowrc on Twitter.

Have a good one!

finis

Update (13 Feb, 2018)

If you’re interested in provisioning TLS certificates, you’re probably also interested in HTTP/2.

I just finished writing a blog post on how to make use of HTTP/2 Server Push with NGINX for those out there making use of NGINX.

Have a good one!