Hey,
There’s been some time since an issue has been open under SwarmKit to address the lack of support to having privileged containers (see https://github.com/docker/swarmkit/issues/1030).
Although pull requests have been made (see https://github.com/docker/swarmkit/pull/1129, for instance), it seems like there’s no intention by the Docker team to have the feature added before entitlements get into moby (see https://github.com/moby/moby/issues/32801).
Knowing those facts, and given that I’ve forked docker before (I blogged before on how to compile and run your own docker fork), why not give a try forking it and add support to privileged containers?
Here’s how the journey looked like.
Let’s get started then!
1. The idea
The whole point of the modifications is as outlined in the swarmkit pull request:
“When create some container, we need
privileged=True. But swarmkit does not support this.”
https://github.com/docker/swarmkit/issues/1030#issue-161106957
That is, the --privileged flag that exists in a regular docker run does not exist in docker service create (and the subsequent API calls used by the CLI).
Having that enables us to have any interesting functionality that requires more privileges than the default ones chosen by the Docker contributors. For instance, with --privileged you can:
- run docker in docker (
dind); - run an NFS server;
- create special devices; and
- more.
As shown in the picture above, there are many places where our privileged annotation has to go on.
Let’s put them in place.
2. Modifying the CLI
Not wanting to interact with the new feature using the remote api directly (using curl, for instance), I decided to start messing with the docker/cli repository.
Here is where we’re able to expose the --privileged flag to users of the docker client. For this part, check out the privileged branch from my fork: https://github.com/cirocosta/docker-cli/tree/privileged
For any Go developer who has already developed an application with flag parsing and all that stuff, the first thing to search for is the place where the definition of the flags to be parsed live.
Given that the docker client has multiple subcommands, the organization in the repository follows the same pattern:
tree ./cli/command -L 1
./cli/command
|-- bundlefile
|-- checkpoint
|-- image
|-- inspect
...
|-- network
|-- node
...
|-- service # <<<<
|-- stack
...
`-- volume
Getting there under service, there’s an opts.go file with all the options that the subcommands of service needs. There you go:
diff --git a/cli/command/service/opts.go b/cli/command/service/opts.go
index 6d427451..b61b6c0f 100644
--- a/cli/command/service/opts.go
+++ b/cli/command/service/opts.go
@@ -483,6 +483,7 @@ type serviceOptions struct {
stopSignal string
+ privileged bool
mounts opts.MountOpt
dns opts.ListOpts
dnsSearch opts.ListOpts
@@ -622,6 +623,7 @@ func (options *serviceOptions) ToService(ctx context.Context, apiClient client.N
Groups: options.groups.GetAll(),
StopSignal: options.stopSignal,
TTY: options.tty,
+ Privileged: options.privileged,
ReadOnly: options.readOnly,
Mounts: options.mounts.Value(),
DNSConfig: &swarm.DNSConfig{
@@ -795,6 +797,8 @@ func addServiceFlags(flags *pflag.FlagSet, opts *serviceOptions, defaultFlagValu
+ flags.BoolVar(&opts.privileged, flagPrivileged, false, "Give extended privileges to the service")
+ flags.SetAnnotation(flagPrivileged, "version", []string{"1.35"})
flags.BoolVarP(&opts.tty, flagTTY, "t", false, "Allocate a pseudo-TTY")
@@ -878,6 +882,7 @@ const (
+ flagPrivileged = "privileged"
flagUpdateDelay = "update-delay"
If we were able to compile cli, then voilà, we’d have --privileged setting a Privileged boolean to true in the service spec.
Why wouldn’t it compile yet? Because the service spec doesn’t have such field! Time to modify SwarmKit.
3. Modifying SwarmKit
Looking at api/specs.proto, we can see the definition of the messages that go through the SwarmKit API.
There we can find one type of message that carries the type of information that fits our purpose: ContainerSpec.
“Container specifies runtime parameters for a container.”
That could only mean one thing - we had to proceed to add a “privileged” to this struct.
Being a property of a protobuf message, that also meant that I’d need to regenerate the Golang definitions that are made up from the protobuf specification. That is, before proceeding, I’d need to grab the protobuf compiler tooling ready.
The installation goes like this:
- grab a
protoc(protobufcompiler) release; and - install the binary and the includes.
So, go to the protobuf releases page and grab the x86-64 (amd64) version.
That’s a zip that comes with the following:
Archive: ./protoc-3.5.1-linux-x86_64.zip
Length Date Time Name
--------- ---------- ----- ----
0 2017-12-22 19:21 include/
0 2017-12-22 19:21 include/google/
0 2017-12-22 19:21 include/google/protobuf/
3781 2017-12-22 19:21 include/google/protobuf/struct.proto
6283 2017-12-22 19:21 include/google/protobuf/type.proto
36277 2017-12-22 19:21 include/google/protobuf/descriptor.proto
7734 2017-12-22 19:21 include/google/protobuf/api.proto
2422 2017-12-22 19:21 include/google/protobuf/empty.proto
0 2017-12-22 19:21 include/google/protobuf/compiler/
8200 2017-12-22 19:21 include/google/protobuf/compiler/plugin.proto
5483 2017-12-22 19:21 include/google/protobuf/any.proto
8196 2017-12-22 19:21 include/google/protobuf/field_mask.proto
3745 2017-12-22 19:21 include/google/protobuf/wrappers.proto
5975 2017-12-22 19:21 include/google/protobuf/timestamp.proto
4890 2017-12-22 19:21 include/google/protobuf/duration.proto
2352 2017-12-22 19:21 include/google/protobuf/source_context.proto
0 2017-12-22 19:21 bin/
4433736 2017-12-21 19:11 bin/protoc
715 2017-12-22 19:21 readme.txt
--------- -------
4529789 19 files
The /bin/protoc is the binary that we need to have in our $PATH, while those protobuf definitions under include must be placed in our default include path (/usr/local/include).
In summary, here’s what you can do:
# Grab the release zip
curl \
-SL \
-o protoc.zip \
https://github.com/google/protobuf/releases/download/v3.5.1/protoc-3.5.1-linux-x86_64.zip
# Unzip it
unzip ./protoc.zip
# Move the correpsonding files
sudo mv ./bin/protoc /usr/local/bin
sudo mv ./include/google /usr/local/include/google
With that, we have protobuf set for generating swarmkit protobuf files.
note.: we don’t need to install github.com/golang/protobuf/protoc-gen-go given that swarmkit uses protoc-gen-gogoswarm.
With that done, we’re not able to apply the changes to our local clone of swarmkit:
--- a/agent/exec/dockerapi/container.go
+++ b/agent/exec/dockerapi/container.go
@@ -209,6 +209,7 @@ func (c *containerConfig) hostConfig() *enginecontainer.HostConfig {
PortBindings: c.portBindings(),
Init: c.init(),
Isolation: c.isolation(),
+ Privileged: c.spec().Privileged,
}
// The format of extra hosts on swarmkit is specified in:
diff --git a/cmd/swarmctl/service/flagparser/container.go b/cmd/swarmctl/service/flagparser/container.go
index 1507a168..37f11dd1 100644
--- a/cmd/swarmctl/service/flagparser/container.go
+++ b/cmd/swarmctl/service/flagparser/container.go
@@ -15,6 +15,15 @@ func parseContainer(flags *pflag.FlagSet, spec *api.ServiceSpec) error {
spec.Task.GetContainer().Image = image
}
+ if flags.Changed("privileged") {
+ privileged, err := flags.GetBool("privileged")
+ if err != nil {
+ return err
+ }
+
+ spec.Task.GetContainer().Privileged = privileged
+ }
+
if flags.Changed("hostname") {
hostname, err := flags.GetString("hostname")
if err != nil {
diff --git a/api/specs.proto b/api/specs.proto
index 14448d04..a5945ec3 100644
--- a/api/specs.proto
+++ b/api/specs.proto
@@ -318,6 +318,9 @@ message ContainerSpec {
// PidsLimit prevents from OS resource damage by applications inside the container
// using fork bomb attack.
int64 pidsLimit = 25;
+
+ // Privileged gives extended privileges to the container
+ bool privileged = 26;
}
protobuf definition updated, now generate the golang protobuf files using swarmkit’s makefile target:
# Install extra dependencies that might be needed.
make setup
# Generate the protobuf golang files.
make generate
# Build the swarkit binaries
make binaries
With everything getting properly compiled, that means that we should now proceed to the next step - incorporating the setting of privileged flag to the ContainerSpec message whenever dockerd receives a Privileged flag in its service spec.
4. Modifying the Docker Daemon
Having a local fork, incorporating the modified swarmkit into the fork doesn’t take much more than setting up the vendoring tool and changing a file here and there.
As docker makes use of vndr to vendor its dependencies, we can modify the vndr configuration pointing swarmkit at our forked repository instead of the regular docker/swarmkit one and have it update the vendor directory with it:
# Get the current commit
swarmkit $ FORKED_SWARMKIT_COMMIT=$(git rev-parse HEAD)
5598a39a89bd7482864c6279ae5544d9e8aa5299
# Go to our forked moby/moby repository
swarmkit $ cd ../docker
# Set a variable that defines the repository for `vndr`
# to use to gather our forked source code.
FORKED_SWARMKIT_REPO=https://github.com/cirocosta/swarmkit
# Prepare the variables to be used in the SED replacement
SED_FROM="\(github.com/docker/swarmkit\) \([a-zA-Z0-9]*\)"
SED_TO="\1 $FORKED_SWARMKIT_COMMIT $FORKED_SWARMKIT_REPO"
# Perform the replacement in `vendor.conf`.
# ps.: you could just grab an editor of your choice
# (like vim) and do the changes there.
#
# Here I'm using `sed` just to make the changes
# reproducible by copying and pasting.
docker $ sed \
--in-place \
"s#$SED_FROM#$SED_TO#g" \
./vendor.conf
# Update the dependencies
vndr
With the dependencies updated (i.e., with our swarmkit code in!), build the binaries (they will now include our swarmkit code):
# Run the `binary` target specified in ./Makefile
make binary
This will make use of a local instance of Docker that you must have running and then generate the daemon binaries at ./bundle:
./bundles/
├── binary-daemon
│ ├── docker-containerd
│ ├── docker-containerd-ctr
│ ├── docker-containerd-ctr.md5
│ ├── docker-containerd-ctr.sha256
│ ├── docker-containerd.md5
│ ├── docker-containerd.sha256
│ ├── docker-containerd-shim
│ ├── docker-containerd-shim.md5
│ ├── docker-containerd-shim.sha256
│ ├── dockerd -> dockerd-dev
│ ├── dockerd-dev
│ ├── dockerd-dev.md5
│ ├── dockerd-dev.sha256
│ ├── docker-init
│ ├── docker-init.md5
│ ├── docker-init.sha256
│ ├── docker-proxy
│ ├── docker-proxy.md5
│ ├── docker-proxy.sha256
│ ├── docker-runc
│ ├── docker-runc.md5
│ └── docker-runc.sha256
└── latest -> .
Naturally, without any changes to docker/docker itself, no flags would be passed to the recently modified SwarmKit ContainerSpec struct.
diff --git a/api/types/swarm/container.go b/api/types/swarm/container.go
index 0041653c9..c16c55fd8 100644
--- a/api/types/swarm/container.go
+++ b/api/types/swarm/container.go
@@ -57,6 +57,7 @@ type ContainerSpec struct {
Privileges *Privileges `json:",omitempty"`
StopSignal string `json:",omitempty"`
TTY bool `json:",omitempty"`
+ Privileged bool `json:",omitempty"`
OpenStdin bool `json:",omitempty"`
ReadOnly bool `json:",omitempty"`
Mounts []mount.Mount `json:",omitempty"`
diff --git a/daemon/cluster/convert/container.go b/daemon/cluster/convert/container.go
index 0a34fc73e..9eb1feb4e 100644
--- a/daemon/cluster/convert/container.go
+++ b/daemon/cluster/convert/container.go
@@ -17,6 +17,7 @@ func containerSpecFromGRPC(c *swarmapi.ContainerSpec) *types.ContainerSpec {
if c == nil {
return nil
}
+
containerSpec := &types.ContainerSpec{
Image: c.Image,
Labels: c.Labels,
@@ -32,6 +33,7 @@ func containerSpecFromGRPC(c *swarmapi.ContainerSpec) *types.ContainerSpec {
OpenStdin: c.OpenStdin,
ReadOnly: c.ReadOnly,
Hosts: c.Hosts,
+ Privileged: c.Privileged,
Secrets: secretReferencesFromGRPC(c.Secrets),
Configs: configReferencesFromGRPC(c.Configs),
Isolation: IsolationFromGRPC(c.Isolation),
@@ -228,6 +230,7 @@ func containerToGRPC(c *types.ContainerSpec) (*swarmapi.ContainerSpec, error) {
Groups: c.Groups,
StopSignal: c.StopSignal,
TTY: c.TTY,
+ Privileged: c.Privileged,
OpenStdin: c.OpenStdin,
ReadOnly: c.ReadOnly,
Hosts: c.Hosts,
diff --git a/daemon/cluster/executor/container/container.go b/daemon/cluster/executor/container/container.go
index 69d673bd3..829c5dcc3 100644
--- a/daemon/cluster/executor/container/container.go
+++ b/daemon/cluster/executor/container/container.go
@@ -351,6 +351,7 @@ func (c *containerConfig) hostConfig() *enginecontainer.HostConfig {
hc := &enginecontainer.HostConfig{
Resources: c.resources(),
GroupAdd: c.spec().Groups,
+ Privileged: c.spec().Privileged,
PortBindings: c.portBindings(),
Mounts: c.mounts(),
ReadonlyRootfs: c.spec().ReadOnly,
Compile it again, and now everything should be fine!
Closing thoughts
To finish the whole thing, update the vendoring of the docker/cli fork to incorporate those changes (yes, docker/cli depends on docker/docker, which depends on docker/swarmkit) and now everything should compile.
Modify your docker.service to use the built binaries, call docker service create using the recently built docker binary and you’re good to go!
If you found any mistakes or felt a little lost, please let me know! I’m cirowrc on Twitter.
Have a good one!
finis