Hey,

There’s been some time since an issue has been open under SwarmKit to address the lack of support to having privileged containers (see https://github.com/docker/swarmkit/issues/1030).

Although pull requests have been made (see https://github.com/docker/swarmkit/pull/1129, for instance), it seems like there’s no intention by the Docker team to have the feature added before entitlements get into moby (see https://github.com/moby/moby/issues/32801).

Knowing those facts, and given that I’ve forked docker before (I blogged before on how to compile and run your own docker fork), why not give a try forking it and add support to privileged containers?

Here’s how the journey looked like.

  1. The idea
  2. Modifying the CLI
  3. Modifying SwarmKit
  4. Modifying the Docker Daemon

Let’s get started then!

Illustration of the journey to get the fork working

1. The idea

The whole point of the modifications is as outlined in the swarmkit pull request:

“When create some container, we need privileged=True. But swarmkit does not support this.”

https://github.com/docker/swarmkit/issues/1030#issue-161106957

That is, the --privileged flag that exists in a regular docker run does not exist in docker service create (and the subsequent API calls used by the CLI).

Having that enables us to have any interesting functionality that requires more privileges than the default ones chosen by the Docker contributors. For instance, with --privileged you can:

  • run docker in docker (dind);
  • run an NFS server;
  • create special devices; and
  • more.
Representation of the process of instantiating a privileged service

As shown in the picture above, there are many places where our privileged annotation has to go on.

Let’s put them in place.

2. Modifying the CLI

Not wanting to interact with the new feature using the remote api directly (using curl, for instance), I decided to start messing with the docker/cli repository.

Here is where we’re able to expose the --privileged flag to users of the docker client. For this part, check out the privileged branch from my fork: https://github.com/cirocosta/docker-cli/tree/privileged

For any Go developer who has already developed an application with flag parsing and all that stuff, the first thing to search for is the place where the definition of the flags to be parsed live.

Given that the docker client has multiple subcommands, the organization in the repository follows the same pattern:

tree ./cli/command -L 1
./cli/command
|-- bundlefile
|-- checkpoint
|-- image
|-- inspect
...
|-- network
|-- node
...
|-- service     # <<<<
|-- stack
...
`-- volume

Getting there under service, there’s an opts.go file with all the options that the subcommands of service needs. There you go:

diff --git a/cli/command/service/opts.go b/cli/command/service/opts.go
index 6d427451..b61b6c0f 100644
--- a/cli/command/service/opts.go
+++ b/cli/command/service/opts.go
@@ -483,6 +483,7 @@ type serviceOptions struct {
        stopSignal      string
+       privileged      bool
        mounts          opts.MountOpt
        dns             opts.ListOpts
        dnsSearch       opts.ListOpts
@@ -622,6 +623,7 @@ func (options *serviceOptions) ToService(ctx context.Context, apiClient client.N
                                Groups:     options.groups.GetAll(),
                                StopSignal: options.stopSignal,
                                TTY:        options.tty,
+                               Privileged: options.privileged,
                                ReadOnly:   options.readOnly,
                                Mounts:     options.mounts.Value(),
                                DNSConfig: &swarm.DNSConfig{
@@ -795,6 +797,8 @@ func addServiceFlags(flags *pflag.FlagSet, opts *serviceOptions, defaultFlagValu
+       flags.BoolVar(&opts.privileged, flagPrivileged, false, "Give extended privileges to the service")
+       flags.SetAnnotation(flagPrivileged, "version", []string{"1.35"})
        flags.BoolVarP(&opts.tty, flagTTY, "t", false, "Allocate a pseudo-TTY")

@@ -878,6 +882,7 @@ const (
+       flagPrivileged              = "privileged"
        flagUpdateDelay             = "update-delay"

If we were able to compile cli, then voilà, we’d have --privileged setting a Privileged boolean to true in the service spec.

Why wouldn’t it compile yet? Because the service spec doesn’t have such field! Time to modify SwarmKit.

3. Modifying SwarmKit

Looking at api/specs.proto, we can see the definition of the messages that go through the SwarmKit API.

There we can find one type of message that carries the type of information that fits our purpose: ContainerSpec.

“Container specifies runtime parameters for a container.”

see swarmkit#api/specs.proto line 164

That could only mean one thing - we had to proceed to add a “privileged” to this struct.

Being a property of a protobuf message, that also meant that I’d need to regenerate the Golang definitions that are made up from the protobuf specification. That is, before proceeding, I’d need to grab the protobuf compiler tooling ready.

Representation of the modification of the swarmkit api protobuf message

The installation goes like this:

  1. grab a protoc (protobuf compiler) release; and
  2. install the binary and the includes.

So, go to the protobuf releases page and grab the x86-64 (amd64) version.

That’s a zip that comes with the following:

Archive:  ./protoc-3.5.1-linux-x86_64.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2017-12-22 19:21   include/
        0  2017-12-22 19:21   include/google/
        0  2017-12-22 19:21   include/google/protobuf/
     3781  2017-12-22 19:21   include/google/protobuf/struct.proto
     6283  2017-12-22 19:21   include/google/protobuf/type.proto
    36277  2017-12-22 19:21   include/google/protobuf/descriptor.proto
     7734  2017-12-22 19:21   include/google/protobuf/api.proto
     2422  2017-12-22 19:21   include/google/protobuf/empty.proto
        0  2017-12-22 19:21   include/google/protobuf/compiler/
     8200  2017-12-22 19:21   include/google/protobuf/compiler/plugin.proto
     5483  2017-12-22 19:21   include/google/protobuf/any.proto
     8196  2017-12-22 19:21   include/google/protobuf/field_mask.proto
     3745  2017-12-22 19:21   include/google/protobuf/wrappers.proto
     5975  2017-12-22 19:21   include/google/protobuf/timestamp.proto
     4890  2017-12-22 19:21   include/google/protobuf/duration.proto
     2352  2017-12-22 19:21   include/google/protobuf/source_context.proto
        0  2017-12-22 19:21   bin/
  4433736  2017-12-21 19:11   bin/protoc
      715  2017-12-22 19:21   readme.txt
---------                     -------
  4529789                     19 files

The /bin/protoc is the binary that we need to have in our $PATH, while those protobuf definitions under include must be placed in our default include path (/usr/local/include).

In summary, here’s what you can do:

# Grab the release zip
curl \
        -SL \
        -o protoc.zip \
        https://github.com/google/protobuf/releases/download/v3.5.1/protoc-3.5.1-linux-x86_64.zip

# Unzip it
unzip ./protoc.zip

# Move the correpsonding files
sudo mv ./bin/protoc /usr/local/bin
sudo mv ./include/google /usr/local/include/google

With that, we have protobuf set for generating swarmkit protobuf files.

note.: we don’t need to install github.com/golang/protobuf/protoc-gen-go given that swarmkit uses protoc-gen-gogoswarm.

With that done, we’re not able to apply the changes to our local clone of swarmkit:

--- a/agent/exec/dockerapi/container.go
+++ b/agent/exec/dockerapi/container.go
@@ -209,6 +209,7 @@ func (c *containerConfig) hostConfig() *enginecontainer.HostConfig {
                PortBindings: c.portBindings(),
                Init:         c.init(),
                Isolation:    c.isolation(),
+               Privileged:   c.spec().Privileged,
        }
 
        // The format of extra hosts on swarmkit is specified in:


diff --git a/cmd/swarmctl/service/flagparser/container.go b/cmd/swarmctl/service/flagparser/container.go
index 1507a168..37f11dd1 100644
--- a/cmd/swarmctl/service/flagparser/container.go
+++ b/cmd/swarmctl/service/flagparser/container.go
@@ -15,6 +15,15 @@ func parseContainer(flags *pflag.FlagSet, spec *api.ServiceSpec) error {
                spec.Task.GetContainer().Image = image
        }
 
+       if flags.Changed("privileged") {
+               privileged, err := flags.GetBool("privileged")
+               if err != nil {
+                       return err
+               }
+
+               spec.Task.GetContainer().Privileged = privileged
+       }
+
        if flags.Changed("hostname") {
                hostname, err := flags.GetString("hostname")
                if err != nil {

diff --git a/api/specs.proto b/api/specs.proto
index 14448d04..a5945ec3 100644
--- a/api/specs.proto
+++ b/api/specs.proto
@@ -318,6 +318,9 @@ message ContainerSpec {
        // PidsLimit prevents from OS resource damage by applications inside the container 
        // using fork bomb attack.
        int64 pidsLimit = 25;
+
+       // Privileged gives extended privileges to the container
+       bool privileged = 26;
 }

protobuf definition updated, now generate the golang protobuf files using swarmkit’s makefile target:

# Install extra dependencies that might be needed.
make setup

# Generate the protobuf golang files.
make generate

# Build the swarkit binaries
make binaries

With everything getting properly compiled, that means that we should now proceed to the next step - incorporating the setting of privileged flag to the ContainerSpec message whenever dockerd receives a Privileged flag in its service spec.

4. Modifying the Docker Daemon

Having a local fork, incorporating the modified swarmkit into the fork doesn’t take much more than setting up the vendoring tool and changing a file here and there.

As docker makes use of vndr to vendor its dependencies, we can modify the vndr configuration pointing swarmkit at our forked repository instead of the regular docker/swarmkit one and have it update the vendor directory with it:

# Get the current commit 
swarmkit $ FORKED_SWARMKIT_COMMIT=$(git rev-parse HEAD)
5598a39a89bd7482864c6279ae5544d9e8aa5299

# Go to our forked moby/moby repository
swarmkit $ cd ../docker

# Set a variable that defines the repository for `vndr`
# to use to gather our forked source code.
FORKED_SWARMKIT_REPO=https://github.com/cirocosta/swarmkit

# Prepare the variables to be used in the SED replacement
SED_FROM="\(github.com/docker/swarmkit\) \([a-zA-Z0-9]*\)"
SED_TO="\1 $FORKED_SWARMKIT_COMMIT $FORKED_SWARMKIT_REPO"

# Perform the replacement in `vendor.conf`.
# ps.: you could just grab an editor of your choice
# (like vim) and do the changes there.
#
# Here I'm using `sed` just to make the changes
# reproducible by copying and pasting.
docker $ sed \
  --in-place \
  "s#$SED_FROM#$SED_TO#g" \
  ./vendor.conf

# Update the dependencies
vndr

With the dependencies updated (i.e., with our swarmkit code in!), build the binaries (they will now include our swarmkit code):

# Run the `binary` target specified in ./Makefile
make binary

This will make use of a local instance of Docker that you must have running and then generate the daemon binaries at ./bundle:

./bundles/
├── binary-daemon
│   ├── docker-containerd
│   ├── docker-containerd-ctr
│   ├── docker-containerd-ctr.md5
│   ├── docker-containerd-ctr.sha256
│   ├── docker-containerd.md5
│   ├── docker-containerd.sha256
│   ├── docker-containerd-shim
│   ├── docker-containerd-shim.md5
│   ├── docker-containerd-shim.sha256
│   ├── dockerd -> dockerd-dev
│   ├── dockerd-dev
│   ├── dockerd-dev.md5
│   ├── dockerd-dev.sha256
│   ├── docker-init
│   ├── docker-init.md5
│   ├── docker-init.sha256
│   ├── docker-proxy
│   ├── docker-proxy.md5
│   ├── docker-proxy.sha256
│   ├── docker-runc
│   ├── docker-runc.md5
│   └── docker-runc.sha256
└── latest -> .

Naturally, without any changes to docker/docker itself, no flags would be passed to the recently modified SwarmKit ContainerSpec struct.

diff --git a/api/types/swarm/container.go b/api/types/swarm/container.go
index 0041653c9..c16c55fd8 100644
--- a/api/types/swarm/container.go
+++ b/api/types/swarm/container.go
@@ -57,6 +57,7 @@ type ContainerSpec struct {
        Privileges      *Privileges             `json:",omitempty"`
        StopSignal      string                  `json:",omitempty"`
        TTY             bool                    `json:",omitempty"`
+       Privileged      bool                    `json:",omitempty"`
        OpenStdin       bool                    `json:",omitempty"`
        ReadOnly        bool                    `json:",omitempty"`
        Mounts          []mount.Mount           `json:",omitempty"`
diff --git a/daemon/cluster/convert/container.go b/daemon/cluster/convert/container.go
index 0a34fc73e..9eb1feb4e 100644
--- a/daemon/cluster/convert/container.go
+++ b/daemon/cluster/convert/container.go
@@ -17,6 +17,7 @@ func containerSpecFromGRPC(c *swarmapi.ContainerSpec) *types.ContainerSpec {
        if c == nil {
                return nil
        }
+
        containerSpec := &types.ContainerSpec{
                Image:      c.Image,
                Labels:     c.Labels,
@@ -32,6 +33,7 @@ func containerSpecFromGRPC(c *swarmapi.ContainerSpec) *types.ContainerSpec {
                OpenStdin:  c.OpenStdin,
                ReadOnly:   c.ReadOnly,
                Hosts:      c.Hosts,
+    Privileged: c.Privileged,
                Secrets:    secretReferencesFromGRPC(c.Secrets),
                Configs:    configReferencesFromGRPC(c.Configs),
                Isolation:  IsolationFromGRPC(c.Isolation),
@@ -228,6 +230,7 @@ func containerToGRPC(c *types.ContainerSpec) (*swarmapi.ContainerSpec, error) {
                Groups:     c.Groups,
                StopSignal: c.StopSignal,
                TTY:        c.TTY,
+    Privileged: c.Privileged,
                OpenStdin:  c.OpenStdin,
                ReadOnly:   c.ReadOnly,
                Hosts:      c.Hosts,
diff --git a/daemon/cluster/executor/container/container.go b/daemon/cluster/executor/container/container.go
index 69d673bd3..829c5dcc3 100644
--- a/daemon/cluster/executor/container/container.go
+++ b/daemon/cluster/executor/container/container.go
@@ -351,6 +351,7 @@ func (c *containerConfig) hostConfig() *enginecontainer.HostConfig {
        hc := &enginecontainer.HostConfig{
                Resources:      c.resources(),
                GroupAdd:       c.spec().Groups,
+               Privileged:     c.spec().Privileged,
                PortBindings:   c.portBindings(),
                Mounts:         c.mounts(),
                ReadonlyRootfs: c.spec().ReadOnly,

Compile it again, and now everything should be fine!

Closing thoughts

To finish the whole thing, update the vendoring of the docker/cli fork to incorporate those changes (yes, docker/cli depends on docker/docker, which depends on docker/swarmkit) and now everything should compile.

Modify your docker.service to use the built binaries, call docker service create using the recently built docker binary and you’re good to go!

If you found any mistakes or felt a little lost, please let me know! I’m cirowrc on Twitter.

Have a good one!

finis