Hey,
There’s been some time since an issue has been open under SwarmKit to address the lack of support to having privileged containers (see https://github.com/docker/swarmkit/issues/1030).
Although pull requests have been made (see https://github.com/docker/swarmkit/pull/1129, for instance), it seems like there’s no intention by the Docker team to have the feature added before entitlements get into moby (see https://github.com/moby/moby/issues/32801).
Knowing those facts, and given that I’ve forked docker before (I blogged before on how to compile and run your own docker fork), why not give a try forking it and add support to privileged containers?
Here’s how the journey looked like.
Let’s get started then!
1. The idea
The whole point of the modifications is as outlined in the swarmkit pull request:
“When create some container, we need
privileged=True
. But swarmkit does not support this.”
https://github.com/docker/swarmkit/issues/1030#issue-161106957
That is, the --privileged
flag that exists in a regular docker run
does not exist in docker service create
(and the subsequent API calls used by the CLI).
Having that enables us to have any interesting functionality that requires more privileges than the default ones chosen by the Docker contributors. For instance, with --privileged
you can:
- run docker in docker (
dind
); - run an NFS server;
- create special devices; and
- more.
As shown in the picture above, there are many places where our privileged
annotation has to go on.
Let’s put them in place.
2. Modifying the CLI
Not wanting to interact with the new feature using the remote api directly (using curl
, for instance), I decided to start messing with the docker/cli repository.
Here is where we’re able to expose the --privileged
flag to users of the docker
client. For this part, check out the privileged
branch from my fork: https://github.com/cirocosta/docker-cli/tree/privileged
For any Go developer who has already developed an application with flag parsing and all that stuff, the first thing to search for is the place where the definition of the flags to be parsed live.
Given that the docker
client has multiple subcommands, the organization in the repository follows the same pattern:
tree ./cli/command -L 1
./cli/command
|-- bundlefile
|-- checkpoint
|-- image
|-- inspect
...
|-- network
|-- node
...
|-- service # <<<<
|-- stack
...
`-- volume
Getting there under service
, there’s an opts.go
file with all the options that the subcommands of service
needs. There you go:
diff --git a/cli/command/service/opts.go b/cli/command/service/opts.go
index 6d427451..b61b6c0f 100644
--- a/cli/command/service/opts.go
+++ b/cli/command/service/opts.go
@@ -483,6 +483,7 @@ type serviceOptions struct {
stopSignal string
+ privileged bool
mounts opts.MountOpt
dns opts.ListOpts
dnsSearch opts.ListOpts
@@ -622,6 +623,7 @@ func (options *serviceOptions) ToService(ctx context.Context, apiClient client.N
Groups: options.groups.GetAll(),
StopSignal: options.stopSignal,
TTY: options.tty,
+ Privileged: options.privileged,
ReadOnly: options.readOnly,
Mounts: options.mounts.Value(),
DNSConfig: &swarm.DNSConfig{
@@ -795,6 +797,8 @@ func addServiceFlags(flags *pflag.FlagSet, opts *serviceOptions, defaultFlagValu
+ flags.BoolVar(&opts.privileged, flagPrivileged, false, "Give extended privileges to the service")
+ flags.SetAnnotation(flagPrivileged, "version", []string{"1.35"})
flags.BoolVarP(&opts.tty, flagTTY, "t", false, "Allocate a pseudo-TTY")
@@ -878,6 +882,7 @@ const (
+ flagPrivileged = "privileged"
flagUpdateDelay = "update-delay"
If we were able to compile cli
, then voilà, we’d have --privileged
setting a Privileged
boolean to true
in the service spec.
Why wouldn’t it compile yet? Because the service spec doesn’t have such field! Time to modify SwarmKit.
3. Modifying SwarmKit
Looking at api/specs.proto
, we can see the definition of the messages that go through the SwarmKit API.
There we can find one type of message that carries the type of information that fits our purpose: ContainerSpec
.
“Container specifies runtime parameters for a container.”
That could only mean one thing - we had to proceed to add a “privileged” to this struct
.
Being a property of a protobuf
message, that also meant that I’d need to regenerate the Golang definitions that are made up from the protobuf
specification. That is, before proceeding, I’d need to grab the protobuf
compiler tooling ready.
The installation goes like this:
- grab a
protoc
(protobuf
compiler) release; and - install the binary and the includes.
So, go to the protobuf releases page and grab the x86-64
(amd64
) version.
That’s a zip that comes with the following:
Archive: ./protoc-3.5.1-linux-x86_64.zip
Length Date Time Name
--------- ---------- ----- ----
0 2017-12-22 19:21 include/
0 2017-12-22 19:21 include/google/
0 2017-12-22 19:21 include/google/protobuf/
3781 2017-12-22 19:21 include/google/protobuf/struct.proto
6283 2017-12-22 19:21 include/google/protobuf/type.proto
36277 2017-12-22 19:21 include/google/protobuf/descriptor.proto
7734 2017-12-22 19:21 include/google/protobuf/api.proto
2422 2017-12-22 19:21 include/google/protobuf/empty.proto
0 2017-12-22 19:21 include/google/protobuf/compiler/
8200 2017-12-22 19:21 include/google/protobuf/compiler/plugin.proto
5483 2017-12-22 19:21 include/google/protobuf/any.proto
8196 2017-12-22 19:21 include/google/protobuf/field_mask.proto
3745 2017-12-22 19:21 include/google/protobuf/wrappers.proto
5975 2017-12-22 19:21 include/google/protobuf/timestamp.proto
4890 2017-12-22 19:21 include/google/protobuf/duration.proto
2352 2017-12-22 19:21 include/google/protobuf/source_context.proto
0 2017-12-22 19:21 bin/
4433736 2017-12-21 19:11 bin/protoc
715 2017-12-22 19:21 readme.txt
--------- -------
4529789 19 files
The /bin/protoc
is the binary that we need to have in our $PATH
, while those protobuf
definitions under include
must be placed in our default include path (/usr/local/include
).
In summary, here’s what you can do:
# Grab the release zip
curl \
-SL \
-o protoc.zip \
https://github.com/google/protobuf/releases/download/v3.5.1/protoc-3.5.1-linux-x86_64.zip
# Unzip it
unzip ./protoc.zip
# Move the correpsonding files
sudo mv ./bin/protoc /usr/local/bin
sudo mv ./include/google /usr/local/include/google
With that, we have protobuf set for generating swarmkit
protobuf files.
note.: we don’t need to install github.com/golang/protobuf/protoc-gen-go
given that swarmkit
uses protoc-gen-gogoswarm
.
With that done, we’re not able to apply the changes to our local clone of swarmkit
:
--- a/agent/exec/dockerapi/container.go
+++ b/agent/exec/dockerapi/container.go
@@ -209,6 +209,7 @@ func (c *containerConfig) hostConfig() *enginecontainer.HostConfig {
PortBindings: c.portBindings(),
Init: c.init(),
Isolation: c.isolation(),
+ Privileged: c.spec().Privileged,
}
// The format of extra hosts on swarmkit is specified in:
diff --git a/cmd/swarmctl/service/flagparser/container.go b/cmd/swarmctl/service/flagparser/container.go
index 1507a168..37f11dd1 100644
--- a/cmd/swarmctl/service/flagparser/container.go
+++ b/cmd/swarmctl/service/flagparser/container.go
@@ -15,6 +15,15 @@ func parseContainer(flags *pflag.FlagSet, spec *api.ServiceSpec) error {
spec.Task.GetContainer().Image = image
}
+ if flags.Changed("privileged") {
+ privileged, err := flags.GetBool("privileged")
+ if err != nil {
+ return err
+ }
+
+ spec.Task.GetContainer().Privileged = privileged
+ }
+
if flags.Changed("hostname") {
hostname, err := flags.GetString("hostname")
if err != nil {
diff --git a/api/specs.proto b/api/specs.proto
index 14448d04..a5945ec3 100644
--- a/api/specs.proto
+++ b/api/specs.proto
@@ -318,6 +318,9 @@ message ContainerSpec {
// PidsLimit prevents from OS resource damage by applications inside the container
// using fork bomb attack.
int64 pidsLimit = 25;
+
+ // Privileged gives extended privileges to the container
+ bool privileged = 26;
}
protobuf
definition updated, now generate the golang protobuf
files using swarmkit
’s makefile target:
# Install extra dependencies that might be needed.
make setup
# Generate the protobuf golang files.
make generate
# Build the swarkit binaries
make binaries
With everything getting properly compiled, that means that we should now proceed to the next step - incorporating the setting of privileged
flag to the ContainerSpec
message whenever dockerd
receives a Privileged
flag in its service spec.
4. Modifying the Docker Daemon
Having a local fork, incorporating the modified swarmkit
into the fork doesn’t take much more than setting up the vendoring tool and changing a file here and there.
As docker
makes use of vndr to vendor its dependencies, we can modify the vndr
configuration pointing swarmkit
at our forked repository instead of the regular docker/swarmkit
one and have it update the vendor
directory with it:
# Get the current commit
swarmkit $ FORKED_SWARMKIT_COMMIT=$(git rev-parse HEAD)
5598a39a89bd7482864c6279ae5544d9e8aa5299
# Go to our forked moby/moby repository
swarmkit $ cd ../docker
# Set a variable that defines the repository for `vndr`
# to use to gather our forked source code.
FORKED_SWARMKIT_REPO=https://github.com/cirocosta/swarmkit
# Prepare the variables to be used in the SED replacement
SED_FROM="\(github.com/docker/swarmkit\) \([a-zA-Z0-9]*\)"
SED_TO="\1 $FORKED_SWARMKIT_COMMIT $FORKED_SWARMKIT_REPO"
# Perform the replacement in `vendor.conf`.
# ps.: you could just grab an editor of your choice
# (like vim) and do the changes there.
#
# Here I'm using `sed` just to make the changes
# reproducible by copying and pasting.
docker $ sed \
--in-place \
"s#$SED_FROM#$SED_TO#g" \
./vendor.conf
# Update the dependencies
vndr
With the dependencies updated (i.e., with our swarmkit
code in!), build the binaries (they will now include our swarmkit code):
# Run the `binary` target specified in ./Makefile
make binary
This will make use of a local instance of Docker that you must have running and then generate the daemon binaries at ./bundle
:
./bundles/
├── binary-daemon
│ ├── docker-containerd
│ ├── docker-containerd-ctr
│ ├── docker-containerd-ctr.md5
│ ├── docker-containerd-ctr.sha256
│ ├── docker-containerd.md5
│ ├── docker-containerd.sha256
│ ├── docker-containerd-shim
│ ├── docker-containerd-shim.md5
│ ├── docker-containerd-shim.sha256
│ ├── dockerd -> dockerd-dev
│ ├── dockerd-dev
│ ├── dockerd-dev.md5
│ ├── dockerd-dev.sha256
│ ├── docker-init
│ ├── docker-init.md5
│ ├── docker-init.sha256
│ ├── docker-proxy
│ ├── docker-proxy.md5
│ ├── docker-proxy.sha256
│ ├── docker-runc
│ ├── docker-runc.md5
│ └── docker-runc.sha256
└── latest -> .
Naturally, without any changes to docker/docker
itself, no flags would be passed to the recently modified SwarmKit ContainerSpec
struct.
diff --git a/api/types/swarm/container.go b/api/types/swarm/container.go
index 0041653c9..c16c55fd8 100644
--- a/api/types/swarm/container.go
+++ b/api/types/swarm/container.go
@@ -57,6 +57,7 @@ type ContainerSpec struct {
Privileges *Privileges `json:",omitempty"`
StopSignal string `json:",omitempty"`
TTY bool `json:",omitempty"`
+ Privileged bool `json:",omitempty"`
OpenStdin bool `json:",omitempty"`
ReadOnly bool `json:",omitempty"`
Mounts []mount.Mount `json:",omitempty"`
diff --git a/daemon/cluster/convert/container.go b/daemon/cluster/convert/container.go
index 0a34fc73e..9eb1feb4e 100644
--- a/daemon/cluster/convert/container.go
+++ b/daemon/cluster/convert/container.go
@@ -17,6 +17,7 @@ func containerSpecFromGRPC(c *swarmapi.ContainerSpec) *types.ContainerSpec {
if c == nil {
return nil
}
+
containerSpec := &types.ContainerSpec{
Image: c.Image,
Labels: c.Labels,
@@ -32,6 +33,7 @@ func containerSpecFromGRPC(c *swarmapi.ContainerSpec) *types.ContainerSpec {
OpenStdin: c.OpenStdin,
ReadOnly: c.ReadOnly,
Hosts: c.Hosts,
+ Privileged: c.Privileged,
Secrets: secretReferencesFromGRPC(c.Secrets),
Configs: configReferencesFromGRPC(c.Configs),
Isolation: IsolationFromGRPC(c.Isolation),
@@ -228,6 +230,7 @@ func containerToGRPC(c *types.ContainerSpec) (*swarmapi.ContainerSpec, error) {
Groups: c.Groups,
StopSignal: c.StopSignal,
TTY: c.TTY,
+ Privileged: c.Privileged,
OpenStdin: c.OpenStdin,
ReadOnly: c.ReadOnly,
Hosts: c.Hosts,
diff --git a/daemon/cluster/executor/container/container.go b/daemon/cluster/executor/container/container.go
index 69d673bd3..829c5dcc3 100644
--- a/daemon/cluster/executor/container/container.go
+++ b/daemon/cluster/executor/container/container.go
@@ -351,6 +351,7 @@ func (c *containerConfig) hostConfig() *enginecontainer.HostConfig {
hc := &enginecontainer.HostConfig{
Resources: c.resources(),
GroupAdd: c.spec().Groups,
+ Privileged: c.spec().Privileged,
PortBindings: c.portBindings(),
Mounts: c.mounts(),
ReadonlyRootfs: c.spec().ReadOnly,
Compile it again, and now everything should be fine!
Closing thoughts
To finish the whole thing, update the vendoring of the docker/cli
fork to incorporate those changes (yes, docker/cli
depends on docker/docker
, which depends on docker/swarmkit
) and now everything should compile.
Modify your docker.service
to use the built binaries, call docker service create
using the recently built docker
binary and you’re good to go!
If you found any mistakes or felt a little lost, please let me know! I’m cirowrc on Twitter.
Have a good one!
finis