This is a “quick” intro to get everyone on the concourse team up to speed on the beginning of our work on containerd - which you can check out the progress in #4783 -, with some of what I learned so far.

Keep in mind that I’m no containerd expert, so I might be stating things that are not 100% true or accurate.


As you probably know, at some point, concourse gets to run definitions of “work to be done” in the form of processes in a machine.

            "I gotta run this thing somewhere"

            - grabs a worker

                    --> hey, create this container following this spec
                    --> btw, run this process in it

Given that the garden interface is high-level enough, as long as the backend that implements it is able to do what’s supposed to, web doesn’t need to worry about who’s implementing it.

    WEB     -----container action-----> Garden implementor (backend)
                                       (how it's called)   <--'

For instance, if you look at the workers that are currently registered against https://ci.concourse-ci.org, you’ll see that we have two different garden implementors: houdini, for windows and darwin, and guardian for the linux ones.

    $ fly -t ci workers
    name                                              containers  platform  tags        team                   
    9561ac54-vm-f742202a-e8d9-45b6-5bbc-7fada8c2c958  4           windows   none        none                   
    c184f9d5-a073-4de7-8ded-e2b78055cc82              4           linux     bosh        main                   
    ci-monitoring-worker-0                            16          linux     none        monitoring-hush-house  
    ci-pr-worker-0                                    3           linux     pr          none                   
    ci-topgun-worker-0                                13          linux     k8s-topgun  none                   
    ci-worker-0                                       53          linux     none        none                   
    darwin-worker                                     4           darwin    none        main                   

As an example of how this looks like in practice, the container creation looks like this (from worker.createGardenContainer):

    func (w workerHelper) createGardenContainer(
            containerSpec ContainerSpec,
            fetchedImage FetchedImage,
            handleToCreate string,
            bindMounts []garden.BindMount,
    ) (gclient.Container, error) {

            // do some setup ...
            env := append(fetchedImage.Metadata.Env, containerSpec.Env...)

            return w.gardenClient.Create(
                            Handle:     handleToCreate,
                            RootFSPath: fetchedImage.URL,
                            Privileged: fetchedImage.Privileged,
                            BindMounts: bindMounts,
                            Limits:     containerSpec.Limits.ToGardenLimits(),
                            Env:        env,
                            Properties: gardenProperties,

And running a process in a container, like this (from worker.gardenWorkerContainer.RunScript, with some code deleted / modified for readability):

    func (container *gardenWorkerContainer) RunScript(ctx context.Context,
            path string, args []string,
            input []byte, output interface{},
            logDest io.Writer, recoverable bool,
    ) error {

            stdout := new(bytes.Buffer)
            stderr := new(bytes.Buffer)

            processIO := garden.ProcessIO{
                    Stdin:  bytes.NewBuffer(input),
                    Stdout: stdout,
                    Stderr: stdout,

            process, err := container.Run(ctx, garden.ProcessSpec{
                    Path: path,
                    Args: args,
            }, processIO)
            if err != nil {
                    return err

            processExited := make(chan struct{})

            go func() {
                    processStatus, processErr = process.Wait()

            select {
            case <-processExited:   // execution finished
                    if processStatus != 0 {
                            return runtime.ErrResourceScriptFailed{
                                    Path:       path,
                                    Args:       args,
                                    ExitStatus: processStatus,
                                    Stderr: stderr.String(),

                    return err

            case <-ctx.Done():       // cancelled
                    return ctx.Err()

That’s all to say that as long as we implement the Garden interface, we can swap the container runtimes as we wish, and that’s exactly the first step that we’re taking with containerd: writing a replacement for guardian in the Linux stack.

    web ...........................................
    .       concourse web ----.
    .                         |
    worker ...................+....................
    .                         |
    .                         |
    .       concourse worker  |
    .               garden backend ---.     :7777
    .                                 |
    .                                 |
    .       containerd                |
    .               /run/containerd/containerd.sock

Again, for web, it’s still communicating with a garden server.


containerd is a container runtime that’s able to manage the complete container lifecycle - from fetching images from a registry, to setting up storage, running, and destroying it. In terms of usage, it’s currently the engine under the hood of moby, buildkit, kubernetes (when using containerd-cri), pouch, and recently, openfaas.

The way that we’re aiming at running it is as a separate process that gets spawn by an ifrit runner that brings it up from the binaries located under the usual Concourse assets path (/usr/local/concourse/bin).

            containerd runner
                    --> /usr/local/concourse/bin/containerd (separate process)

Once up, the interaction with it takes place through a client that has all that we need to touch a running containerd instance in our machine (the worker):

    client, err := containerd.New("/run/containerd/containerd.sock")
    if err != nil {
            err = fmt.Errorf("containerd client conn: %w", err)

    defer client.Close()

Under the hood, this takes cares of instantiating the grpc client and “dialing” to the unix socket, so that further interactions with containerd can take place throguh remote procedure calls that we perform against it.

    // New returns a new containerd client that is connected to the containerd
    // instance provided by address
    func New(address string, opts ...ClientOpt) (*Client, error) {
            gopts := []grpc.DialOption{
                    grpc.WithBackoffMaxDelay(3 * time.Second),

            connector := func() (*grpc.ClientConn, error) {
                    ctx, cancel := context.WithTimeout(context.Background(), copts.timeout)
                    defer cancel()
                    conn, err := grpc.DialContext(ctx, dialer.DialAddress(address), gopts...)
                    if err != nil {
                            return nil, errors.Wrapf(err, "failed to dial %q", address)
                    return conn, nil

            conn, err := connector()
            if err != nil {
                    return nil, err

            // ...

That’s because differently from garden where the interface is a REST-a-like HTTP-based API, containerd expose its functionality through grpc services whose specs are defined in the form of protocol buffers in its interface definition language.

                    containerd client

                            ------grpc rpc ----> containerd

For instance, looking at the “containers” protobuf spec, we can see the definition of how one one interact with the provider of the “containers service”.

    syntax = "proto3";

    service Containers {
            rpc Get(GetContainerRequest) returns (GetContainerResponse);
            rpc List(ListContainersRequest) returns (ListContainersResponse);
            rpc ListStream(ListContainersRequest) returns (stream ListContainerMessage);
            rpc Create(CreateContainerRequest) returns (CreateContainerResponse);
            rpc Update(UpdateContainerRequest) returns (UpdateContainerResponse);
            rpc Delete(DeleteContainerRequest) returns (google.protobuf.Empty);

    message Container {
            string id = 1;
            map<string, string> labels  = 2;

            // ...

            google.protobuf.Timestamp created_at = 8 [(gogoproto.stdtime) = true, (gogoproto.nullable) = false];

    message GetContainerRequest {
            string id = 1;

    message GetContainerResponse {
            Container container = 1 [(gogoproto.nullable) = false];

(from containers.proto)

ps.: for a simple “hello world” example of grpc-go, check out cirocosta/hello-grpc.

Using the gRPC toolchain, the containerd maintainers take that definition and turn it into both client and server-side code that implements it, which we can then consume (the client) from the github.com/containerd/containerd package.

    containers, err := client.ContainerService().List(context.Background()))
    if err != nil {
            err = fmt.Errorf("list containers: %w", err)

    fmt.Printf("%-16s %s", "ID", "CREATED-AT")
    for _, container := range containers {
            fmt.Printf("%-16s %s",

That said, as long as we’re matching compatible versions of the server (containerd) and the client (the github.com/containerd/containerd Go package), that interface is “guaranteed” to be honored.


    module github.com/concourse/concourse

    require (
            github.com/containerd/containerd v1.3.2
            // ...

    go 1.13


    RUN curl -sSL $URL/releases/download/v$CONTAINERD_VERSION.linux-amd64.tar.gz \
            | tar -zvxf - -C /usr/local/concourse/bin --strip-components=1 && \


If we try to run that code above though, it wouldn’t work - we’re missing something: namespaces.

Whenever interacting with the containerd API, we must specify against which “tenant” we’re interacting with, allowing containerd to be targetted by multiple consumers without conflicts between the objects maintained for each of them.

      | `` what are the containers
      |           in NS1?         ,,
      '------>      containerd
              |  NS1          |   NS2              |
              |  containers   |  other containers  |
              |  some imgs    |  some other imgs   |
              |   ...         |   ..               |

To fix that example above then, we could either get the namespace information into the context used for that call, or configure the client with a default namespace for all calls to containerd made by that client.

    ctx := namespaces.WithNamespace(context.Background(), "ns1")

    // or

    client, err := containerd.New(


Internally, containerd is composed of multiple components, whose interfaces are exposed as gRPC services, which are presented in a nice high-level form through the containerd API that the client interacts with (through gRPC).

    concourse containerd backend   |
    ----------------------------   | client
          containerd client        |
                 | grpc
          containerd API           |
    ----------------------------   | server
       low-level grpc services     |

Despite these being very loosely coupled, they’re meant to work together from the desire of the higher level containerd API.

For instance, in the process of going from having an image in a registry, to actually running a container, quite few of those need to interact:

            --> fetch
                    --> content
                    --> images

            --> unpack
                    <-- content
                    <-- images
                    --> snapshots

ps.: having this decoupled nature, some of this components are actually swappable by custom plugins - e.g., you can bring your Snapshotter (storage), or Task (runtime).

In this first iteration of our Garden backend though, to have the least amount of divergences from what we currently run on top of (guardian), what I proposed was that we don’t use containerd fetcher, and instead, continue leveraging baggageclaim for now, which already gives us a root filesystem that we can use in our containers.

    task to run
            --> baggaclaim prepares a volume for it
            --> garden references that volume
                    --> containerd uses that rootfs volume

By doing this, we can learn from all of the pieces that we’ll definitely get wrong, and have a more “oranges to oranges” comparison when it comes to its day-to-day operations.


For running a container, few services are usually involved, just like for images.

Assuming that we’re using containerd’s image fetcher, the flow would look like the following:

                    <-- images
                    --> snapshot
                    <-- snapshot
                    --> containers  (metadata)
                    <-- containers
                    --> tasks       *actual container

First, it starts by reading the image’s configuration, creating an OCI spec that describes the container that we want to run, then create a cow layer to serve as the rootfs.

Once that’s done, it then moves on to setting up the linux namespaces, mounts, etc, and then actually starting the process itself.

In our case though, if we assume that baggageclaim is there to give us the volumes, and that we don’t run containers “to completion”, but rather, we use them as “places to execute stuff”, it looks more like this:

    1. garden.Create(containerSpec)

                    --> containers  (creates metadata that specifies
                                     that we want a container)

                    <-- containers
                    --> task        * the container

    2. garden.Run(processSpec, processIO)

                    <-- task
                    --> process     * our process in the task sandbox

Despite containerd being the thing that provides the API for dealing with all of the lifecycle of a container, the containers themselves are not tied to the lifecycle of containerd - there’s a strict separation between those.

When it comes the time to run a container, it forks off a shim which is responsible for calling out to runc, these two being re-parented by the system’s init (they decouple themselves from containerd).


As the shim expects is tailored towards a specific version of runc, that’s another dependency that we must also get the right version.