Docker multi-arch with buildx.

I use docker a lot, like… all the time, I use it locally and I run it on my servers, I use it for job and even in private.
When working, I prefer to “bring my own images” to make sure that the images are not tampered with and that I know what is going on in them. Hence, most base images that are used within Jitesoft are built from scratch with debian, ubuntu or alpine linux as the initial images.

During my PKI explorations I encountered an issue where I wanted to host a few containers on an ARM machine. Why is not too important while the more important question is: why didnt it work?!
The answer is quite easy, and I did know it before, even if I didnt think about it. All my images where at that point only built for AMD64 CPUs, none else. This was something that had to change!

Earlier, to build docker images for multiple architectures required that you built multiple images, created a shared manifest, pushed each image and combined them as a final image by using the manifest and removed the old images in the registry… well, yeah, that is quite an annoying task…
Entering: Docker BuildX, or rather, BuildKit.

Buildx is just a name used to show that it is “experimental”, it will not be a plugin with a special name when it is finally complete.

Buildkit allows us to use some magic things, such as… build multiple images with a single builder, a builder that creates the manifest and even pushes it all to the registries you tag it for!
Further on, by using the experimental syntax in your docker images, you have access to even more special features!

In this post, I won’t go through how to use the experimental syntax, that is a later thing, I will just go through how to set up docker buildx and how to create a multi-arch builder.

Step 1, Installing buildx

If you use the latest docker version (anything after 19.03.2) you might already have the buildx plugin built in to the docker cli. So we start with enabling it and check if it’s there, if not, we install it.

There are two parts required to be able to use buildx and all the features from it. One is to enable the experimental features in the CLI client, and the second is to enable the experimental features in the daemon.
We start with the daemon.

Open (create if it does not exist) your ~/.docker/config.json file.
Inside that file, add the following property:

{
  "experimental": "enabled"
}

And that’s it. Now it should be enabled. Check if it is by typing docker version. You should see something like this:

Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:29:11 2019
 OS/Arch:           linux/amd64
 Experimental:      true

Where the last line will indicate if the experimental features are on or off.

To enable it in the daemon, you should open the /etc/docker/daemon.json file and add the following property:

{
  "experimental": true
}

As you might notice, the first file used "enabled" while the second used true… Don’t ask me why, I really wish I knew!
Restart the docker daemon (service docker restart) and type docker version again. The output (daemon part) should look something like the following:

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:27:45 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

The experimental part under engine should now show true instead of false and it is enabled!

When this is done, we can check if buildx is bundled in your version, type docker help | grep buildx, which should output something like this: buildx* Build with BuildKit (Docker Inc., v0.3.0-5-g5b97415-tp-docker)

If it does, you have the buildx plugin in docker already and can skip the installation of the plugin part (if you are not interested!).

Install BuildX plugin

Docker cli plugins is a quite new thing, they are simple and they are very easy to add to docker. The first thing we do is to fetch the latest version of buildx, which can be found at: https://github.com/docker/buildx/releases we then place the file inside the ~/.docker/cli-plugins directory and makes it runnable. And done!

Short convenience script:

BUILDX_VERSION=<latest version>
mkdir -p ~/.docker/cli-plugins
curl -L  https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64 -o ~/.docker/cli-plugins/docker-buildx

Retry docker help | grep buildx to make sure that it’s there and we go on to the next step!

Step 2, emulate architectures!

To run multi-arch builds on your AMD computer you need to be able to emulate other architectures. Easiest way to do this is to use QEMU, a great emulator for stuff like this! Installing it can be done manually or by using dockers pre-built image which will do it for you:

Check https://hub.docker.com/r/docker/binfmt/tags?page=1&ordering=last_updated to know which tag you should use, I always use the newest.

docker run --rm --privileged docker/binfmt:66f9012c56a8316f9244ffd7622d7c21c1f6f28d

And now you have all the files needed to run multi-arch!

Step 3, set up a builder.

Creating a new builder is quite simple, but there are a few things that might be good to know. In its current version, you will want to use the docker-container type of builder to be able to push the images directly, to build multi platform images and to be able to use the cache export features. It’s the default when creating a new, but to be sure, I always state that I want it to be one either way (not like its a huge thing to write!).

We COULD specify what arch’s to use when we create the builder, but to make sure that only the ones that are possible to use on the given computer, I usually leave this to the bootstrapping phase to decide.

docker buildx create --name my-new-builder --driver docker-container --use
docker buildx inspect --bootstrap

The above command creates a new builder, set it to default/current and inspects it (with bootstrap flag). When inspecting a new buildx container will be started and create some files in your .docker directory with information about what builder is the current, and such.
When the bootstrapping process is finished, you can run docker buildx ls to check your builders, the new one should show that you can use it to build on more platforms than the default one!

Step 4, build a multi platform image!

When building an image with multiple platform targets, you might in some cases require to know what the building platform is and what the target platform is, there are a few arguments that are injected when running buildx build, and those are the following:

TARGETPLATFORM # Platform of the target, for example: 'linux/arm64' or `linux/arm/v7` 
TARGETOS       # Os part of the target platform, for example: 'linux' or 'windows'
TARGETARCH     # Arch part of the target platform, for example 'amd64', 'arm64' or 'arm'
TARGETVARIANT  # If the target platform have a variant (`v7` in arm for example), this will be set to that value
BUILDPLATFORM  # The platform of the build machine
BUILDOS        # OS of the build machine
BUILDARCH      # Architecture of the build machine
BUILDVARIANT   # Variant (if exist) of the build machine

As usual, you should define those with the ARG directive in your image, else they will not be available to use.

As an example, we could create a very very simple image to make x-arch:

FROM jitesoft/alpine
# (this ^ image is already x-arch, so can be used as a base if wanted!)
RUN apk add --no-cache curl

ENTRYPOINT ["curl"]
CMD ["--version"]

To build this for multiple architectures, we use buildx:

docker buildx build --platform=linux/amd64,linux/arm64,linux/s390x --push -t my-org/my-image:latest .

The above command will build the image for amd64, arm64 and s390x as linux containers, it will tag the result to my-org/my-image:latest and push it to the registry (in default case docker hub).
That’s really it, basically…
There are of course a lot more that can be done and a whole lot more that you will encounter as issues and as lovely new features to docker, but this post includes the very basics on it and should allow you to get started right away!