Signing OCI Images with Cosign!

Cosign is a fairly new (v1 release 28 July 2021) project which is a part of the sigstore project to ease signing, storing and verifying signatures for Container images.

As a vivid user of PGP and other signature solutions, cosign is something I’ve been looking at for a while. I just recently started using it to sign container images to be able to implement a bit more strict procedure in my container building and usage flow, something that was a lot easier than I thought!

So, I think that part covers at the least the ‘What’ bit of the excerpt, so let’s head on to ‘Why’.

Signatures and Validation

Most people working in tech have encountered signatures from time to time, it’s often a hazzle if you wish to verify stuff and to keep your own keys can be a real pain. The sigstore project have a few solutions to ease the process (fulcio), but this part will focus more on the signing and verifying bits.

When it comes to why someone would like to sign their images (or binaries or whatever they feel like signing), the reason is quite simple, it’s to allow for consumers to validate the resource and (as long as you are someone they trust) know that what they download, run or install is actually from you and is safe to use.

With my company, I try to make sure that we are both transparent and a source of trustworthy software, and when we release things, I want people to be able to verify that it’s actually from us, not from someone using the name or even hijacking one of our accounts.
Signing and verification consists of two keys and a checksum, the private key is owned by the one who produces the resource, while the public key is to allow validation. The checksum that is the actual signature only fits the private key and if it’s signed with the wrong key, the public key will not accept it as a valid signature.

That way, the trust is in that the key is safe and not used by anyone but the developer while any binary uploaded anywhere can be validated to at the least that point.

There are occurrences when a key runs aloft and malicious software is published with a valid signature, but it’s a lot rarer than seeing an access token or similar get lost and used.

Keeping a “root” key secret is not always easy and depending on the risk of a lost key, there are a lot of different approaches one could take.

I would personally always recommend that one create their root key on a non-online machine and that the machine is wiped after generating the key and some intermediates, but when it comes to this tutorial, we will just use a fresh key and think about the paranoia later!

How

The first thing one have to do to be able to sign their images with cosign is to get ahold of the software.
Cosign can either be installed or ran through docker (or another container runtime). I will describe the ‘install’ way here rather than using docker.

Installation

If you are a go user (which cosign, just as about every new non-frontend-web project now ‘a day is!) and have go 1.6+ installed, you can install the latest version of cosign with a simple go install command:

go install github.com/sigstore/cosign/cmd/cosign@latest

But if you prefer to use binaries, the easiest way is to download the binary from GitHub:

(Linux-ish way!)

# Depending on your architecture you might want to use something other than amd64
ARCH=amd64
LATEST=$(wget -qO- https://api.github.com/repos/sigstore/cosign/releases | jq -r ".[0].name")
wget https://github.com/sigstore/cosign/releases/download/${VERSION}/cosign-linux-${ARCH}
mv cosign-linux-${ARCH} /usr/bin/cosign
chmod +x /usr/bin/cosign

For Windows user, there are exe releases on the cosign release page on GitHub: https://github.com/sigstore/cosign/releases

Test to make sure that it’s installed correctly

cosign version
GitVersion:    v1.2.1
GitCommit:     unknown
GitTreeState:  unknown
BuildDate:     unknown
GoVersion:     go1.16.6
Compiler:      gc
Platform:      linux/amd64

And initialize cosign (creates a .sigstore config directory in your home dir): cosign init.

Create keys!

Now when we have cosign installed we can actually create our first key-pair.
This is simply done by invoking the cosign binary like this:

cosign generate-key-pair

Which produces a cosign.key and a cosign.pub file in the directory where you ran it.
The private key is supposed to be private, really private. If you lose your private key and someone gets a hold of it, they can basically upload binaries with valid signatures in your name, something you really don’t want!

With that said, we can’t throw the key on a usb stick and forget about it for the next 100 years, no, we actually need the key to sign our images!

The public key is for your users. You can distribute it basically however you want, it’s public and totally okay to just throw into a gist or to upload on a warez site! No one can do much with it more than validating your payloads anyway!

So, when we got our two keys, we can basically start sign our images right away.

To sign an image, you use the cosign sign -key cosign.key my-org/my-image, this will create a new signature and upload it to the registry.
To test if it’s a valid image, you use the cosign verify -key cosign.pub my-org/my-image and you should get an output which looks something like this:

Verification for index.docker.io/jitesoft/ubuntu:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
  - Any certificates were verified against the Fulcio roots.

[{"critical":{"identity":{"docker-reference":"index.docker.io/jitesoft/ubuntu"},"image":{"docker-manifest-digest":"sha256:e2700dee042c018ed9505940f6ead1de72155c023c8130ad18cd971c6bfd4f03"},"type":"cosign container image signature"},"optional":{"sig":"jitesoft-bot"}}]

With a lill bit of JQ, you can get it pretty printed as well!:

Verification for index.docker.io/jitesoft/ubuntu:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
  - Any certificates were verified against the Fulcio roots.
[
  {
    "critical": {
      "identity": {
        "docker-reference": "index.docker.io/jitesoft/ubuntu"
      },
      "image": {
        "docker-manifest-digest": "sha256:e2700dee042c018ed9505940f6ead1de72155c023c8130ad18cd971c6bfd4f03"
      },
      "type": "cosign container image signature"
    },
    "optional": {
      "sig": "jitesoft-bot"
    }
  }
]

As you can see in the payload above, there is an ‘optional’ object in the JSON in which I have added a ‘sig’ key. That’s called an annotation, with annotations you can add any arbitrary data to the signature layer pushed to the registry.
Adding annotations is done by the -a flag (can be used multiple times) and then a key=value as the flag value.

Automate it

As many others, I don’t manually build my images, I use CI scripts on GitLab. To sign each and every new image published I’d have to spend most of my days signing images, something I do not like to do, so I want to have it as a part of my build script. The sigstore project supplies a cosign action for GitHub, which can be easily used by a single step:

uses: sigstore/cosign-installer@main

I don’t use GitHub for my pipelines though, so I decided to write my own template for GitLab.
The template can be easily extended from the Jitesoft gitlab-ci-template library if wanted, or you could copy it and modify it after your own choice, as most of the stuff I do outside of client work, it’s released under the MIT license.

The following script is the one that I use:

.sign:
  image: registry.gitlab.com/jitesoft/dockerfiles/cosign:latest
  variables:
    COSIGN_ANNOTATIONS: ""
  before_script:
    - if [ -z ${COSIGN_PUB_KEY_PATH+x} ]; then echo "Failed to find public key"; exit 1; fi
    - if [ -z ${COSIGN_PRIV_KEY_PATH+x} ]; then echo "Failed to find private key"; exit 1; fi
    - |
      if [ ! -z ${DOCKER_CRED_FILE+x} ]; then
        mkdir ~/.docker
        cp ${DOCKER_CRED_FILE} ~/.docker/config.json
      fi
    - |
      if [ ! -z ${SIGN_IMAGES+x} ] && [ ! -z ${SIGN_TAGS} ]; then
        wget https://gist.githubusercontent.com/Johannestegner/093e8053eabd795ed84b83e9610aed6b/raw/helper.sh
        chmod +x helper.sh
        COSIGN_TAGS=$(./helper.sh imagelist "${SIGN_IMAGES}" "${SIGN_TAGS}")
      elif [ -z ${COSIGN_TAGS+x} ]; then
        echo "Failed to find tags to sign"
        exit 1
      fi
  script:
    - |
      for IMAGE in ${COSIGN_TAGS};
      do
        cosign sign ${COSIGN_ANNOTATIONS} -key ${COSIGN_PRIV_KEY_PATH} ${IMAGE}
      done

OBSERVE if you run this script, only allow it on protected branches, and if possible, make sure you use your own runners. And as I have said many times in this post… DO NOT LOSE YOUR PRIVATE KEY!

As you can see, I do a bit more than just sign the image, so I’ll briefly explain how it works.

The image used is an image built (and signed of course!) under the jitesoft organisation.
It’s based on alpine linux to allow for a small-ish image but still the ability to run stuff interactively (the official cosign image is a distroless image).
To skip using a docker run in the configuration, I use the image as the actual image the scripts run in.

It verifies that there are two environment variables (which I personally use gitlab secrets for) are set (should point to the path of the public and private key, even though the public key is not currently used) and then, if there is a docker credentials json file, it moves it to the home folder of the non-root user.

To make it even more easy for myself, I decided to include a small helper script in case the SIGN_IMAGES and SIGN_TAGS variables are set, it basically loops through all the images and gives them the same tags on all (as you can see in the helper.sh execution in the script).

When the tags are set up (or there already is a COSIGN_TAGS variable set) the script moves on to actually signing the tags with the cosign sign command. Any optional annotations is included with the COSIGN_ANNOTATIONS variable.

Final words

I hope that this little tutorial gave some insight on why one would want to use cosign and similar tools and also how to use it in a simple way.

I will keep on using cosign and update or create a new post about it in the future, especially when I start testing fulcio in a larger scale!