• 1 - HA k3s provisioning
• 2 - HA k3s - Networking in Hetzner
• 3 - HA k3s - Deploying the network

As stated in the first post in this series, my plan is to deploy the cluster with the help of CI/CD pipelines in gitlab.
Even if you don’t plan to deploy from a pipeline, some parts of this post could be useful still, as I will write about states, and especially remote states.

I’ll try to use the TF/tf abbreviation for terraform and opentofu, as they both use that themselves. The projects are still quite close to each other in functionality, but when it comes to the GitLab implementation, it’s OpenTofu that is used.

If you use Terraform, the encryption part of this guide might not work as intended and you might want to leave any auto encryption out. The encryption clause is new to OpenTofu.

TF in GitLab

GitLab have quite good support for Terraform and OpenTofu both as a state storage as well as a module registry, further, they supply ci/cd components to ease setting up deployment of the projects, which is really nice.

TF state

When you deploy a tf project, a state is created, the state describes the currently deployed infrastructure.
To allow sharing of states (excluding sending files), one of the easiest and best ways are to use a remote state storage.

There are a bunch of storage types, but in this case, I decided to use the one that GitLab provides (as that’s where I have my files and pipelines).

To tell gitlab to use their storage as your remote state storage, you aught to add a new object to the terraform clause.
In my case, I added a new terraform file named backend.tf which initially looks like this:

terraform {
  backend "http" {}
}

On the tf side, that is basically all you have to do for gitlab to create the state in their storage.
Now, that does not mean that we are done, we need a pipeline!

As I earlier mentioned, gitlab provides a set of components, and to make it super easy, we will use the full-pipeline version.

The .gitlab-ci.yml file should look like this:

include:
  - component: $CI_SERVER_FQDN/components/opentofu/full-pipeline@0.50.0
    inputs:
      version: 0.50.0
      opentofu_version: 1.9.0
      auto_encryption: true
      auto_encryption_passphrase: $ENCRYPTION_PASSPHRASE

stages: [validate, build, deploy, cleanup]

The components are versioned, so I would expect the 0.50.0 component to keep on working, but I would recommend checking the Component repository and see if anything new have been added.

The above pipeline will run the following jobs:

Stage validate:

• fmt (will run a tofu format command)
• validate (will run a tofu validate command)

Stage build:

• plan (will run a tofu plan command)

Stage deploy:

• apply (will run a tofu apply command)

Stage cleanup:

• destroy (will run a tofu destroy command and tear down the infra)
• clean-state (will delete the state stored in the remote state storage)

The Deploy and Cleanup stages are both ‘Manual’, which means that you will have to actively invoke them in the GitLab UI for them to run (which is quite good, seeing you don’t want to accidentally delete your infrastructure!).

As you can see in the pipeline file, there are a few inputs which are set:

version is the component version, from my understanding it is used for the sub-components, and should for now be set, while there is an issue in the gitlab tracker to make it use the value from the initial component inclusion string.

opentofu_version is the version of the OpenTofu executable. As of writing this, 1.9.0 is the latest, but to make sure the version you choose is correct, take a look in the component repository.

Now to the important stuff…

auto_encryption is a boolean value which tells gitlab to include a TF_ENCRYPT variable, which in turn activates automatic encryption of the state file.
This is likely something you will want to do, but it’s important to know that you need to save your encryption passphrase somewhere safe, so that you can decrypt the state in case something goes wrong.

auto_encryption_passphrase is the passphrase used to encrypt the state, I use a GitLab variable which is protected, masked and hidden.

Deploy the first version

Seeing we use some custom provider passwords and such, we need to create a tfvars file that the deployment can use, a tfvars file is a simple key-value file which we define variable values in (seeing gitlab isn’t interactive!).

The file should look something like this:

hcloud_token = "my-hcloud-token-created-in-previous-post"

Now, we don’t really need to save this file anywhere, but as we will want a variable file later on, we might as well create a dev.tfvars file in the root of the project and add it to .gitignore.

The text in the file should be added to gitlab though.
We don’t want to commit a secret, so for this, we use the Variables and mark it as a file.

Name the Key to GITLAB_TOFU_VAR_FILE and gitlab will pick it up in the commands!

When this is done, we can push the repository to the main branch and check the pipelines…

Press apply and your network will be deployed to Hetzner!

State and Validate…

Now, the component we use currently (as of 20250113) have an issue…
When the state file is encrypted and the validate command runs, it will not use the backend at all, this means that it can’t decrypt the state, hence not download the Hetzner provider.

I have yet found a good way to get around this, so in my CI pipeline, I added a rule to never run the validate job:

include:
  - component: $CI_SERVER_FQDN/components/opentofu/full-pipeline@0.50.0
    inputs:
      # ... 
      validate_rules:
        - when: never

With this change, the validate job won’t run and no failure will happen.

Access the state locally

When we work with tf, it’s quite often that we want to make sure that the plan is possible to apply
(As you probably know, this is done with the terraform|tofu plan command).
To plan locally, you need the state. But now our state is encrypted and placed in a remote location…

To allow the state to be downloaded and decrypted in your local environment, there are two things that have to be done:

Encryption in the terraform backend

In our backend.tf file, we need to add a new clause called encryption.

terraform {
  backend "http" {}

  encryption {
    key_provider "pbkdf2" "gitlab_tofu_auto_encryption" {
      passphrase = var.gitlab_state_encryption_passphrase
    }

    method "aes_gcm" "gitlab_tofu_auto_encryption" {
      keys = key_provider.pbkdf2.gitlab_tofu_auto_encryption
    }

    state {
      enforced = true
      method   = method.aes_gcm.gitlab_tofu_auto_encryption
    }

    plan {
      enforced = true
      method   = method.aes_gcm.gitlab_tofu_auto_encryption
    }
  }
}

We define the key_provider, which is a pbkdf2 (a PSK crypto function) and the same as the one GitLab currently uses, it should basically just set the passphrase property to the passphrase value.
In my case, I added the passphrase to my local tfvars file and the variables.tf file I use to expose it.

The next part is the method used to decrypt the state, this uses aes_gcm, which is an AES based gcm algorithm, the same as the one GitLab currently uses, the keys parameter is set to the key_provider we created above.

We then set the state and plan objects to enforce encryption and to use the method we defined.

Initialize tf with backend config

When this is done, we need to init the tf project with a lengthy command, the full command can be found in gitlab under the Operate > Terraform States tab, press the vertical ... under Actions and select the Copy Terraform init command.

The command looks like this:

export GITLAB_PROJECT_ID=<PROJECT-ID>
export GITLAB_ACCESS_TOKEN=<YOUR-ACCESS-TOKEN>
export TF_STATE_NAME=default
tofu init \
    -backend-config="address=https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/terraform/state/$TF_STATE_NAME" \
    -backend-config="lock_address=https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/terraform/state/$TF_STATE_NAME/lock" \
    -backend-config="unlock_address=https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/terraform/state/$TF_STATE_NAME/lock" \
    -backend-config="username=<your-username>" \
    -backend-config="password=$GITLAB_ACCESS_TOKEN" \
    -backend-config="lock_method=POST" \
    -backend-config="unlock_method=DELETE" \
    -backend-config="retry_wait_min=5"

After this have been invoked, your local state will be updated from the remote state, now we can try tofu plan and see that the state is downloaded and decrypted.

Final words

And that’s it. We now have a way to deploy our network to hetzner with a CI/CD pipeline! Kinda nice and easy eh?

There are additional things that can be (and I will try to write about here) done with the pipeline, one thing I really want to add to my own pipeline is to do a tofu plan on pull requests and display the infra difference in the PR directly.

As usual, let me know if you find any oddities in the post and I’ll correct it asap!

• 1 - HA k3s provisioning
• 2 - HA k3s - Networking in Hetzner
• 3 - HA k3s - Deploying the network

In this post, I’ll go through how to set up a new network in Hetzner cloud with the help of terraform.
This is usually a good first step on the road to a cluster, seeing that without a network, we would have to use external traffic for all communication, and even if hetzner is very generous with the egress limits on the VM:s, it feels quite dumb.

So, to start of we need to get a hold of an API key for the hetzner provider.

Hetzner project and API key

When you first enter the hetzner cloud dashboard, you will be able to create a new project.
The API key we will use is connected to a single project, so it won’t have access to your other projects, seeing the key is specific for the project, that is all that is needed to actually provision machines.

After creating the project, select ‘Security’ in the sidebar, toggle to ‘API tokens’ tab and generate a new token.
Make sure you select “read/write” rather than just read, else the Token won’t allow terraform to actually provision anything.

Terraform providers

As you probably know, you can use either Terraform or OpenTofu for provisioning. Either tool is good, and you should choose the one fitting your preferences. Still, I will be using terraform in most commands in the series, which, if you use OpenTofu, just requires you to swap terraform to tofu in the commands.

After creating our token, we can start setting up our tf project.

The initial file structure could look something like this:

network/
  _providers.tf
  main.tf
  variables.tf

Inside the providers.tf file, we add the setup code for the providers we will be using. In this case, just the hetzner provider:

terraform {
  required_providers {
    hcloud = {
      source = "hetznercloud/hcloud"
      version = "1.49.1"
    }
  }
}

provider "hcloud" {
  token = var.hcloud_token
}

Make sure you check the terraform registry and change the version of the provider accordingly.

As you might see, we are using a var in the hcloud provider. This variable is something we will add to the variables.tf file, and force the “user” to provide on provisioning:

variable "hcloud_token" {
  sensitive = true
}

The sensitive flag in the variable will hide the token from terraform output, while it might be worth knowing that it will not hide it from the state file. So if you intend to share state, I would recommend that you use the ephemeral flag which was recently introduced to terraform.

When this is done, we can initialize the project:

terraform init
# or
tofu init

This will download the provider from the registry and allow us to use it.

Add a network

We will use one big network for the cluster, while all different resources will be using different subnets, this to make sure we can set up firewalls accordingly and make the IP-addresses a bit nicer to look at ;)

The first part we need to consider is the size of the network.
If you want to read up on how an ip CIDR works, feel free to read my post about this here.

In my case, I prefer to use a /16 network for my project, which will give me 65536 internal IP addresses.
This might feel quite a bit over the top, and surely it is, but each network can pretty much use a full private ip-range, so rather go big than too low!

Add the following code to the variables file:

variable "core_cidr" {
  type = string
  default = "10.1.0.0/16"
  description = "CIDR used by the core network"
  
  validation {
    condition = try(
      "" != var.core_cidr,
      regex("(\\d{1,3}[.]\\d{1,3}[.]\\d{1,3}[.]\\d{1,3}[/]\\d{1,2})", var.core_cidr)
    )
    error_message = "Must be a valid cidr / subnet (example: 10.1.0.0/16)."
  }
}

The above configuration will give us a IP range between 10.1.0.0 to 10.1.255.255.

The variable above contains quite a bit more stuff than when we added the token, this is because I prefer it to actually be assignable on the provisioning step (I usually go with the default value, but I might want to use another network later on).

The big difference is the validation clause, which takes the variable and makes sure its actually a valid-ish CIDR. As you may see (if you know regex a bit), i’ve been lazy, and 999.999.999/99 would be allowed in the regex, so if you want more safety, make sure to update the regex!

Main.tf

Our next step is to add the network resource to the main.tf terraform file.

This is quite a simple resource with just a few required parameters, and it looks like this:

resource "hcloud_network" "core" {
  name     = "my-core-network"
  ip_range = var.core_cidr

  labels = {
    "usage" = "kubernetes"
    "identifier"  = "core"
  }
}

We want to use a specific name here, as we will want to be able to import the network as a datasource later on, and I have as well added a few labels to the resource to make sure I can select on it if I don’t want to use the name.

With this, we actually have everything we need to create a network in the project at hetzner.

You can plan and apply the resource with the following commands:

terraform plan # Will show you what will be created in hetzner.
terraform apply # Will actually provision the resources.

Subnets

As I earlier mentioned, we will want a few subnets in the network.
The following networks are the ones that I will be using:

Master nodes: 10.1.10.0/27 (10.1.10.0 - 10.1.10.31), 32 IP addresses.
Minion nodes: 10.1.20.0/23 (10.1.10.0 - 10.1.11.255) 512 IP addresses.
Misc: 10.1.5.0/24 (10.1.5.0 - 10.1.5.255) 256 IP addresses.

These networks can be increased when we need to, and are currently just using a very small part of the internal network.
As you can see, there are 32 ip addresses in the master nodes network (and excluding gateway and such only 30), but I don’t think that my cluster will ever use more than 30 master nodes, and if that happens, I will increase the size!

The Misc network will be used for stuff like load balancers and other resources that we might be needing later on.

Set up master subnet

I don’t want to jump ahead too far yet, but we might as well set up the master subnet right away, so that it is ready for when we want to provision our master nodes…

A subnet in hetzner is just another resource to be provisioned, and it looks as the following:

resource "hcloud_network_subnet" "master-net" {
  ip_range     = var.master_cidr
  network_zone = "eu-central"
  type         = "cloud"
  network_id   = hcloud_network.core.id
}

In the current form, I decided to put it in the same main.tf file as the core network, but if you want to split it up, you can retrieve the core network as a datasource:

data "hcloud_network" "core" {
  name = "my-core-network"
}

resource "hcloud_network_subnet" "master-net" {
  ip_range     = var.master_cidr
  network_zone = "eu-central"
  type         = "cloud"
  network_id   = data.hcloud_network.core.id
}

Just as with the core network CIDR, we will want to add the master network CIDR to the variables file:

variable "master_cidr" {
  type = string
  default = "10.1.10.0/27"

  description = "IP Address range for Master nodes"

  validation {
    condition = try(
      "" != var.master_cidr,
      regex("(\\d{1,3}[.]\\d{1,3}[.]\\d{1,3}[.]\\d{1,3}[/]\\d{1,2})", var.master_cidr)
    )
    error_message = "Must be a valid cidr / subnet (example: 10.1.10.0/27)."
  }
}

When this is done, we can run the apply command again, and then check the hetzner dashboard to make sure our network exists.

Outputs

In my project, I have split up my project into multiple repositories. This is not really required, but due to this, I need to be able to access values from the Network project in other projects as well. This requires me to generate an Output which is saved to the state, allowing me to import the state of the network project with a terraform_remote_state datasource in the other projects.

Outputs are values “outputed” from the module, in my case, the ones I need to output are the subnets (as they are not “real” resources in hcloud).
I want these so that I can put the resources in their correct subnets without me having to copy the strings to either project.

The outputs file looks as following:

output "subnets" {
  value = {
    "minion-cidr" : hcloud_network_subnet.minion-net.ip_range
    "master-cidr" : hcloud_network_subnet.master-net.ip_range
    "misc-cidr" : hcloud_network_subnet.misc-net.ip_range
  }
}

output "core-net" {
  value = {
    cidr = hcloud_network.core.ip_range
    id   = hcloud_network.core.id
  }
}

Final thoughts

Using terraform is quite simple when it comes to smaller projects, but as it grows, it gets more and more important to set up a good file structure as well as structure inside the files.
When we later on add Ansible playbooks and various templates and files to the project, this will become more and more obvious.

As always, let me know in the comments below if you find any oddities or issues!

A long time ago I wrote an (incomplete) post series about setting up k8s with the help of ansible and terraform on the UpCloud platform.
The world have moved forward quite a bit, and I have for a while been using hetzner cloud to host my development/staging cluster.
I write this series to give myself a bit of an incentive to actually complete both the series and the cluster setup, something I have been postponing for quite a few years…

Disclaimer: I’m not affiliated with Hetzner or any other company which is referred to in this series.

My current cluster is manually provisioned and uses k3s (which is a slimmed down version of kubernetes), which works quite alright, but it’s not HA and I would really prefer to be able to re-provision it easily (both the cluster itself and all resources running in it). So, the new cluster I want to build should be automatically provisioned and modified via CI (GitLab), HA (3x master nodes) and for now, running on hetzner.
In the future, other providers could be added to make it multi-cloud, but I don’t need that right now.

The reason I go with hetzner for this, is because they have good and cheap shared vcpu instances with both AMD64 and ARM64 architecture.
Their API and terraform provider is great, and they are located in the EU, which is one of my requirements.
Further, if you require more power, they have dedicated cpu instances as well as bare metal machines.
Their other services covers most of the standard stuff you would want from a budget cloud provider.

So, what are my plans?

Well, I’m quite fond of terraform, so that is what I will be using to provision the VM:s in hetzner.
The software on the machines should as well be automatically installed, but in this case I’m less sure.
I’m most comfortable with Ansible when it comes to that, so It will most likely be the tool of choice.

This post will be updated with links to all the posts of the project, and I’ll try to make as much open source as possible.

To get the most out of this series, you should have some fundamental understanding on how terraform and ansible works, as well as an account at Hetzner for doing laborations.
You can always bring down the cluster with a terraform destroy at any time, so costs can be cut quite low if you don’t have a promo code.

If you wish to use my promo code (gives $20 in free credits and if you decide to keep on using hetzner it will give me some credits too), feel free to use the following link for signup: Referral link! (make sure you read the terms before signing up).

This post focuses on Windows 11 and WSL2, this is because the updates to WSL required to mount disks like this does not exist for Windows 10 nor.

Initialize the VHDX

The first step is to create a virtual drive. This can be done with the help of HyperV or through PowerShell:

New-Vhd -Dynamic -SizeBytes 20gb -BlockSizeBytes 1mb -Path C:\path\to\my-new-disk.vhdx

This will initialize a new dynamic disk which will grow up to 20gb, the BlockSizeBytes param tells the VHD to increase size 1mb at a time, this can be changed to your liking, but its the default in HyperV.

Depending on the user whom created the disk, you might have to change the permissions in the properties of the VHDX file to allow your user to do everything.

The new disk is not formatted, so we need to mount it into wsl to allow our linux distro (in this example Ubuntu) to format it.
So on the first mount, we use

wsl --mount --vhd C:\path\to\my-new-disk.vhdx --bare

Start wsl and check the disks with lsblk.

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0 388.4M  1 disk
sdb      8:16   0     8G  0 disk [SWAP]
sdc      8:32   0    20G  0 disk

In my case, the disk I want to use is sdc. This might be hard to know if you have multiple disks mounted, so you might want to unmount the disk, run lsblk and then diff the output when re-mounting.

When you know which disk it is you want to format, it’s time to format it.
Depending on filesystem, there are a few different commands, but in this post, I’ll use ext4.

sudo mkfs.ext4 /dev/sdc

After this is done, the disk will be formatted and you should unmount it from wsl:

wsl --unmount --vhd C:\path\to\my-new-disk.vhdx

If you don’t care about automatically mounting the disk, and just want to do it manually each time, you can just use the following command:

wsl --mount --vhd C:\path\to\my-new-disk.vhdx --name <disk-name>

And then locate the disk in /mnt/wsl/<disk-name> inside your wsl installation.

Automount the VHDX

For this part, we will create a small script and make it run with the TaskScheduler, this is quite a simple thing to do but I for one have not used the Scheduler very much, so it might be new for you as well.

The script will run the mount command and a simple mount-wsl.cmd file will do just fine.
Create the file in a location that feels good and add the following snippet:

wsl --mount --vhd C:\path\to\my-new-disk.vhdx --name <disk-name>

If you got more disks than one, just add the same line with another disk name and path.

Next up is the scheduler, which you can find by pressing win + r and typing in taskschd.msc.
Right-click on the Task Scheduler Library and select Create Task.

Name the task to an appropriate name and add some type of description to remind you of what it does (not like we visit the scheduler that often!). The following boxes should be checked under the General tab:

• Run only when user is logged on
• Run with highest privileges

The next tab to go to is the Trigger tab, here you should press New to create a new trigger.

We want the trigger to happen on login for our specific user, so change the Begin the task dropdown to At log on and toggle Specific user.
Press Ok and the trigger will be created.

The final thing we need to do is to create the action that will be triggered, this is done in the Action tab.
The action to use is Start a program (which is the only non-deprecated action), and the Program/Script we want to run is our mount-wsl.cmd script.

Select it with the Browse button and press Ok, no arguments needed.

If your on a desktop computer, you’re done. Save the Task and press Run to see if it works as intended.
If your on a laptop, there is one last thing you might want to change:

In the Conditions tab, there are two checkboxes:

Start the task only if the computer is on AC power and Stop if the computer switches to battery power.
It’s likely that you wish to change those two, seeing the task in question is very low power and a fire-and-forget task.

And that’s it.

Final words

This post is mainly intended as a reminder to myself (as I had to re-do it on a new computer just a bit ago), while it could be useful for others as well, so hope you enjoyed it!

I really like the ease of setting it up, and the fact that they support a wide array of database engines makes their operators very useful.

In this post, we will focus on XtraDB, which is their mysql version with backup and clustering capabilities.
We will go through the installation of the operator as well as what I find most important in the custom resource which will allow us to provision a full XtraDb cluster with backups and proxying.
This is what I’ve been using the most, and I’ll try to create a post at a later date with some benchmarks to show how it compares with other databases.

Running databases in kubernetes (or docker) have earlier been a big no-no, this is not as much of an issue now ‘a days, especially when using good storage types.
In this writeup, I’ll use my default storage class which on my k3s cluster is a mounted disk on Hetzner, they are decent in speed, but seeing it’s just a demo, the speed doesn’t matter much!

Prerequisites

Percona xtradb makes use of cert-manager to generate TLS certificated, it will automatically create an issuer (namespaced) for your resources, but you do need to have cert-manager installed.
This post will not cover the installation, and I would recommend that you take a look at the official cert-manager doucmentation for installation instructions.

Helm installation

The first thing we have to do is to install the helm charts and deploy them to a kubernetes cluster.
In this post, we will as I said earlier, use the Mysql version of percona, and we will use the operator that is supplied by percona.

If you want to dive deeper, you can find the documentation here!

Percona supplies their own helm charts for the operator via GitHub, so adding it to helm is easily done with

helm repo add percona https://percona.github.io/percona-helm-charts/
helm repo update

If you haven’t worked with helm before, the above snippet will add it to your local repository and allow you to install charts from the repo we add.

If you just want to install the operator right away, you can do this by invoking the helm install command, but we might want to look a bit at the values we can pass to the operator first, to customize it slightly.

The full chart can be found on GitHub, where you should be able to see all the customizable values in the values.yml file (the ones set are the default values).
In the case of this operator, the default values are quite sane, it will create a service account and set up the RBAC values required for it to monitor the CRD:s.
But, one thing that you might want to change is the value for watchAllNamespaces.
The default value here is false, which will only allow you to create new clusters in the same namespace as the operator. This might be a good idea if you have multiple tenants in the cluster, and you don’t want all of them to have access to the operator, while for me, making it a cluster-wide operator is far more useful.

To tell the helm chart that we want it to change the said value, we can either pass it directly in the install command, or we can set up a values file for our specific installation.
When you change a lot of values, or you want to source-control your overrides, a file is for sure more useful.

To create an override file, you need to create a values.yml (you can actually name it whatever you want) where you set the values you want to change, the format is the same as in the above repository values.yml file, so if we only want to change the namespaces parameter it would look like this:

watchAllNamespaces: true

But any value in the default values file can be changed.

Installing the operator with the said values file is done by invoking the following command:

helm install percona-xtradb-operator percona/pxc-operator -f values.yml --namespace xtradb --create-namespace

The above command will install the chart as percona-xtradb-operator in the xtradb namespace.
You can change namespace as you wish, and it will create the namespace for you.
If you don’t want the namespace to be created (using another one or default) skip the --create-namespace flag.
Without using the namespace flag, the operator will be installed in the default namespace.
The file we changed is passed via the -f flag, and will override any values already defined in the default values file.

When we set the watchAllNamespaces value, the helm installation will create cluster wide roles and bindings, this does not happen if it’s not set but is required for the operator to be able to look for and manage clusters in all namespaces.

If you don’t want to use a custom values file, passing values to helm is done easily by the following flags:

helm install percona-xtradb-operator percona/pxc-operator --namespace xtradb --set watchAllNamespaces=true

Multi Arch clusters

Currently, the operator images (and other as well) are only available for the AMD64 architecture, so in cases where you use nodes which are based on another architecture (like me who use a lot of ARM64), you might want to set the nodeSelector value in your override to only use amd64 nodes:

nodeSelector:
  kubernetes.io/arch: amd64

To update your installation, instead of using install (which will make helm yell about already having it installed) you use the upgrade command:

helm upgrade percona-xtradb-operator percona/pxc-operator -f values.yml --namespace xtradb

If you are lazy like me, you can actually use the above command with the --install flagg to install as well.

Percona xtradb CRD:s

As with most operators, the xtradb operator comes with a few custom resource definitions to allow easy creation of new clusters.

We can install a new db cluster with helm as well, but I prefer to version control my resources and I really enjoy using the CRD:s supplied by operators I use, so we will go with that!

So, to install a new percona xtradb cluster, we will create a new kubernetes resource as a yaml manifest.

The cluster uses the api version pxc/percona.com/v1 and the kind we are after is PerconaXtraDBCluster.
There are a lot of configuration that can be done, and a lot you really should look deeper into if you are intending to run the cluster in production (especially the TLS options and how to encrypt the data at rest).
But to keep this post under a million words, I’ll focus on the things we need to just get a cluster up and running!

As with all kubernetes resources, we will need a bit of metadata to allow kubernetes to know where and what to create:

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  name: cluster1-test
  namespace: private

In the above manifest, I’m telling kubernetes that we want a PerconaXtraDBCluster set up in the private namespace using the name cluster1-test.
There are a few extra finalizers we can add to the metadata to hint to the operator how we want it to handle removal of clusters, the ones that are available are the following:

• delete-pods-in-order
• delete-pxc-pvc
• delete-proxysql-pvc
• delete-ssl

These might be important to set up correctly, as they will allow for the operator to remove PVC:s and other configurations which we want it to remove on cluster deletion.
If you do want to save the claims and such, you should not include the finalizers in the metadata.

Sepc

After the metadata have been set, we want to start working on the specification of the resource.
There is a lot of customization tha can be done in the manifest, but the most important sections are the following:

• tls (which allows us to use cert-manager to configure mTLS for the cluster)
• upgradeOptions (which allows us to set up upgrades of the running mysql servers)
• pxc (the configuration for the actual percona xtradb cluster)
• haproxy (configuration for the HAProxy which runs in front of the cluster)
• proxysql (configuration for the ProxySQL instances in front of the cluster)
• logcollector (well, for logging of course!)
• pmm (Percona monitor and management, which allows us to monitor the instances)
• backup (this one you can probably guess the usage for!)

TLS

In this writeup I will leave this with the default values (and not even add it to the manifest), that way the cluster will create its own issuer and just issue tls certificates as it needs to, but if you want the certificate to be a bit more constrained, you can here set boh which issuer to use (or create) as well as the SANs to use.

UpgradeOptions

Keeping your database instances up to date automatically is quite a sweet feature. Now, we don’t always want to do this seeing we sometimes want to use the exact same version in the cluster as in another database (if we got multiple environments for example) or if we want to stay on a version we know is stable.
But, if we want to live on the edge and use the latest version, or stay inside a patch version of the current version we use this section is very good.

There are three values that can be set in the upgradeOptions section, and they handle the scheduling, where to look and the version constraints we want to sue.

upgradeOptions:
  versionServiceEndpoint: ' https://check.percona.com'
  apply: '8.0-latest'
  schedule: '0 4 * * *'

The versionServiceEndpoint flag should probably always be https://check.percona.com, but if there are others you can probably switch. I’m not sure about this though, so to be safe, I keep it at the default one!

apply can be used to set a constraint or disable the upgrade option all together.
If you don’t want your installations to upgrade, just set it to disabled, then it will not run at all.
In the above example, I’ve set the version constraint to use the latest version of the 8.0 mysql branch.
This can be set to a wide array of values, for more detailed info, I recommend checking the percona docs.

The Schedule is a cron-formatted value, in this case, at 4am every day, to continuously check, set it to * * * * *!

pxc

The pxc section of the manifest handles the actual cluster setup.
It got quite a few options, and I’ll only cover the ones I deem most important to just run a cluster, while as said earlier, if you are intending to run this in production, make sure you check the documentation or read the CRD specification for all available options.

spec:
  pxc:
    nodeSelector:
      kubernetes.io/arch: amd64
    size: 3
    image: percona/percona-xtradb-cluster:8.0.32-24.2
    autoRecovery: true
    expose:
      enabled: false
    resources:
      requests:
        memory: 256M
        cpu: 100m
      limits:
        memory: 512M
        cpu: 200m
    volumeSpec:
      persistentVolumeClaim:
        storageClassName: 'hcloud-volumes'
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi

The size variable will tell the operator how many individual mysql instances we want to run.
3 is a good one, seeing most clustered programs prefer 3 or more instances.

The image should presumably be one of the percona images in this case, to allow updates and everything to work as smoothly as possible.
I haven’t peeked enough into the images, but I do expect that there is some custom things in the images to make them run fine, which makes me want to stick to the default images rather than swapping to another!

autoRecovery should probably almost always be set to true, this will allow the Automatic Crash Recovery functionality to work, which I expect is something most people prefer to have.

I would expect that you know how resources works in kubernetes, but I included it in the example to make sure that its seen, as you usually want to be able to set those yourself. The values set above are probably quite a bit low when you want to be able to use the database for more than just testing, so set them accordingly, just remember that it’s for each container, not the whole cluster!

The volumeSpec is quite important. In the above example, I use my default volume type, which is a RWO type of disk the size is set to 10Gi. The size should probably either be larger or possible to expand on short notice.

There are two more keys which can be quite useful if you wish to customize your database a bit more, and especially if you want to finetune it.
Percona xtradb comes with quite sane defaults, but when working with databases, it’s not unusual that you need to enter some custom params to the my.cnf file.

Environment variables

The percona pxc configuration does not currently allow bare environment variables (from what I can see), but this is not a huge issue, seeing the spec allows for a envVarsSecret to be set.
The secret must of course be in the same namespace as the resources, but any variables in it will be loaded as environment variables into the pod.
I’m not certain what environment variables are available for the pxc section, but will try to update this part when I got more info on it.

Configuration

The configuration property expects a string, the string is a mysql configuration file, i.e, the values that you usually put in the my.cnf file.

spec:
  pxc:
    configuration: |
      [mysqld]
      innodb_write_io_threads = 8
      innodb_read_io_threads = 8

HAProxy and ProxySQL

Percona allows you to choose between two proxies to use for loadbalancing, which is quite nice.
The available proxies are HAProxy and ProxySQL, both valid choices which are well tried in the industry for loadbalancing and proxying.

The one you choose should have the property enabled set to true, and the other one set to false.

The most “default” configuration you can use would look like this:

# With haproxy
spec:
  haproxy:
    nodeSelector:
      kubernetes.io/arch: amd64
    enabled: true
    size: 3
    image: percona/percona-xtradb-cluster-operator:1.13.0-haproxy
    resources:
      requests:
        memory: 256M
        cpu: 100m
# With proxysql
spec:
  proxysql:
    nodeSelector:
      kubernetes.io/arch: amd64
    enabled: true
    size: 3
    image: percona/percona-xtradb-cluster-operator:1.13.0-proxysql
    resources:
      requests:
        memory: 256M
        cpu: 100m
    volumeSpec:
      emptyDir: {}

The size should be at the least 2 (can be set to 1 if you use allowUnsafeConfigurations but that’s not recommended).
The image is just as with the pxc configuration most likely best to use the percona provided images (in this case 1.13.0, same version as the percona operator).

As always, the resources aught to be finetuned to fit your needs, the above is on the lower end, but could work okay for a smaller cluster which does not have huge traffic.

Both of the sections allow for (just as with pxc section) to supply both environment variables via the envVarsSecret as well as a configuration property. The configuration does of course differ and I would direct you to the proxy documentation for more information about those!
Now, something quite important to note here is that if you supply a configuration file, you need to supply the full file, it doesn’t merge the default file but replaces it in full.
So if you want to finetune the configuration, include the default configuration as well (and change it), this applies to both haproxy and proxysql and will work the same if you use a configmap, secret or directly accessing the configuration key.

The choice of proxy might be important to decide on at creation of the resource, if you use proxysql, you can (with a restart of the pods) switch to haproxy, while if you choose haproxy, you can’t change the cluster to use proxysql. So I would highly recommend that you decide which to use before creating the cluster.

There are a lot more variables you can set here, and all of them can be found at the documentation page.

LogCollector

Logs are nice, we love logs! Percona seems to as well, because they supply us with a section for configuring a fluent bit log collector right in the manifest! No need for any sidecars, just turn it on and start collecting :)

If you already have some type of logging system which captures all pods logs and such, this might not be useful and you can set the enabled value to false and ignore this section.

The log collector specification is quite slim, and looks something like this:

spec:
  logcollector:
      enabled: true
      image: percona/percona-xtradb-cluster-operator:1.13.0-logcollector
      resources:
        requests:
          memory: 64M
          cpu: 50m
      configuration: ...

The default values might be enough, but the fluent bit documentation got quite a bit of customization available if you really want to!

PPM (Monitoring)

The xtradb server is able to push metrics and monitoring data to a PMM (percona monitoring & management) service, now, this is not installed with the cluster and needs to be set up separately, but if you want to make use of this (which I recommend, seeing how important monitoring is!) the documentation can be found here.

I haven’t researched this too much yet, but personally I would have loved to be able to scrape the instances with prometheus and have my dashboards in my standard Grafana instance, which I will ask percona about if it’s possible. In either case, I’ll update this part with more information when I figure it out!

Backups

Backups, one of the most important parts of keeping a database up and running without angry customers questioning you about where their 5 years of data has gone after a database failure… Well, percona helps us with that too, thankfully!

The percona backup section allows us to define a bunch of different storage engines to use to store our backups, this is great, because we don’t always want to store our backups on the same disks or systems as we run our cluster. The most useful way to store backups is likely to store them in a s3 compatible storage, which can be done, but if you really want to you can store them either in a PV or even on the local disk of the node. We can even define multiple storages to use with different schedules!

spec:
  backup:
  image: perconalab/percona-xtradb-cluster-operator:main-pxc8.0-backup
  storages:
    s3Storage:
      type: 's3'
      nodeSelector:
        kubernetes.io/arch: amd64
      s3:
        bucket: 'my-bucket'
        credentialsSecret: 'my-credentials-secret'
        endpointUrl: 'the-s3-service-i-like-to-use.com'
        region: 'eu-east-1'
    local:
      type: 'filesystem'
      nodeSelector:
        kubernetes.io/arch: amd64
      volume:
        persistentVolumeClaim:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10G
  schedule:
    - name: 'daily'
      schedule: '0 0 * * *'
      keep: 3
      storageName: s3Storage
    - name: 'hourly'
      schedule: '0 * * * *'
      keep: 2
      storageName: 'local'

In the above yaml, we have set up two different storage types. One s3 type and one filesystem type.
The s3 type is pointed to a bucket in my special s3-compatible storage while the filesystem one makes use of a persistent volume.

In the schedule section, we set it to create a daily backup to the s3 storage (and keep the 3 latest ones) while the local storage one will keep 3 and run every hour.

Each section under storages will spawn a new container, so we can change the resources and such for each of them (and you might want to) and they will by default re-try creation of the backup 6 times (can be changed by setting the spec.backup.backoffLimit to a higher value).

There is a lot of options for backups, and I would highly recommend to take a look at the docs for it!

Point in time

One thing that could be quite useful when working with backups for databases is point in time recovery.
Percona xtradb have this available in the backup section under the pitr section:

spec:
  backup:
    pitr:
      storageName: 'local'
      enabled: true
      timeBetweenUploads: 60

It makes use of the same storage as defined in the storages section, and you can set the interval on PIT uploads.

Restoring a backup

Sometimes our databases fails very badly, or we get some bad data injected into it. In cases like those we need to restore an earlier backup of said database.
I won’t cover this in this blogpost, as it’s too much to cover in a h4 in a tutorial like this, but I’ll make sure to create a new post with disaster scenarios and how percona handles them.

If you really need to recover your data right now (before my next post) I would recommend that you either read the Backup and restore and “How to restore backup to a new kubernetes-based environment” section in the documentation.

The full chart

Now, when we have had a look at the different sections, we can set up our full chart:

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  name: cluster2-test
  namespace: private
spec:
  upgradeOptions:
    versionServiceEndpoint: ' https://check.percona.com'
    apply: '8.0-latest'
    schedule: '0 4 * * *'
  pxc:
    size: 3
    nodeSelector:
      kubernetes.io/arch: amd64
    image: percona/percona-xtradb-cluster:8.0.32-24.2
    autoRecovery: true
    expose:
      enabled: false
    resources:
      requests:
        memory: 256M
        cpu: 100m
      limits:
        memory: 512M
        cpu: 200m
    volumeSpec:
      persistentVolumeClaim:
        storageClassName: 'hcloud-volumes'
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
  haproxy:
    enabled: true
    nodeSelector:
      kubernetes.io/arch: amd64
    size: 3
    image: percona/percona-xtradb-cluster-operator:1.13.0-haproxy
    resources:
      requests:
        memory: 256M
        cpu: 100m
  proxysql:
    enabled: false
  logcollector:
    enabled: true
    image: percona/percona-xtradb-cluster-operator:1.13.0-logcollector
    resources:
      requests:
        memory: 64M
        cpu: 50m
  backup:
    image: perconalab/percona-xtradb-cluster-operator:main-pxc8.0-backup
    storages:
      s3Storage:
        type: 's3'
        nodeSelector:
          kubernetes.io/arch: amd64
        s3:
          bucket: 'my-bucket'
          credentialsSecret: 'my-credentials-secret'
          endpointUrl: 'the-s3-service-i-like-to-use.com'
          region: 'eu-east-1'
      local:
        type: 'filesystem'
        nodeSelector:
          kubernetes.io/arch: amd64
        volume:
          persistentVolumeClaim:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10G
    schedule:
      - name: 'daily'
        schedule: '0 0 * * *'
        keep: 3
        storageName: s3Storage
      - name: 'hourly'
        schedule: '0 * * * *'
        keep: 2
        storageName: 'local'
    pitr:
      storageName: 'local'
      enabled: true
      timeBetweenUploads: 60

Now, to get the cluster running, just invoke kubectl and it’s done!

kubectl apply -f my-awesome-cluster.yml

It takes a while for the databases to start up (there are a lot of components to start up!) so you might have to wait a few minutes before you can start play around with the database.
Check the status of the resources with the get kubectl command:

kubectl get all -n private


NAME                                   READY   STATUS    RESTARTS   AGE
pod/cluster1-test-pxc-0                3/3     Running   0          79m
pod/cluster1-test-haproxy-0            2/2     Running   0          79m
pod/cluster1-test-haproxy-1            2/2     Running   0          78m
pod/cluster1-test-haproxy-2            2/2     Running   0          77m
pod/cluster1-test-pxc-1                3/3     Running   0          78m
pod/cluster1-test-pxc-2                3/3     Running   0          76m

NAME                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                 AGE
service/cluster1-test-pxc                ClusterIP   None            <none>        3306/TCP,33062/TCP,33060/TCP            79m
service/cluster1-test-pxc-unready        ClusterIP   None            <none>        3306/TCP,33062/TCP,33060/TCP            79m
service/cluster1-test-haproxy            ClusterIP   10.43.45.157    <none>        3306/TCP,3309/TCP,33062/TCP,33060/TCP   79m
service/cluster1-test-haproxy-replicas   ClusterIP   10.43.54.62     <none>        3306/TCP                                79m

NAME                                     READY   AGE
statefulset.apps/cluster1-test-haproxy   3/3     79m
statefulset.apps/cluster1-test-pxc       3/3     79m

When all StatefulSets are ready, you are ready to go!

Accessing the database

When a configuration as the one above is applied, a few services will be created.
The service you most likely want to interact with is called <your-cluster-name>-haproxy (or -proxysql depending on proxy) which will proxy your commands to the different backend mysql servers.
From within the cluster it’s quite easy, just accessing the service, while outside will require a loadbalancer service (which can be defined in the manifest) alternatively a ingress which can expose the service to the outer net.

If you wish to test your database from within the cluster, you can run the following command:

kubectl run -i --rm --tty percona-client --namespace private --image=percona:8.0 --restart=Never -- bash -il
percona-client:/$ mysql -h cluster1-haproxy -uroot -proot_password

The root password can be found in the <your-cluster-name>-secrets secret under the root key.

Final words

I really enjoy using percona xtradb, it allows for really fast setup of mysql clusters with backups enabled and everything one might need.
But, I’m quite new to the tool, and might have missed something vital or important!
So please, let me know in the comments if something really important is missing or wrong.

As you might know, I’m an ambassador for Civo, a cloud native service provider. Since about a year ago they started to host conferences, where the first one was in the US, and now in september, the first EU one was hosted in London at The Brewery, and I was invited.

I’ve earlier written about my issues with anxiety, something which makes traveling to another country for a couple of days kinda hard. So to make it easier for me, I decided to bring my 10-year-old son (I was assured that it would be child-friendly enough before hehe).
I’ll try not to dwell in this kind of issues in this post, seeing it’s quite irrelevant, but I can tell you that it all went very well (with a tiny issue where I bought a ticket with train to Maregate instead of Moorgate and almost ended up way off course!).

The brewery and our first day in London

The Brewery was more than I expected! It was a really hot period in London while we were there (above 30c every day, in September!!), but the ventilation at the brewery was so good that I even had to wear my hoodie during some of the talks!
Over the two days of the conference, food (which I gathered was made at The Brewery) was served. A lot of it was stuff I (nor my son) had not tried before, and it tasted fantastic.
The areas for the talks where of great size and the staff succeeded greatly in both sound and light during all talks we attended.
The rooms used for workshops where large enough while still cozy.

The brewery have quite a history (with brewing of beer if you can imagine), something I would recommend checking out at their site if you are interested in brewing history!

We decided to pre-register the day before the event started, and got to meet some of the Civo Staff, whom where very friendly and easy to talk to, we then took a small stroll through Islington and ended up eating sushi at Itsu, a place which ended up being the only place my son wanted to eat (he really loves sushi).

We headed back to the hotel quite early and watched a movie to be able to get up early for the keynotes.

Day one of the conference

We succeeded in getting up at a decent time to grab some breakfast at the hotel before heading out.
We were both a tad bit sad that there was no real “English Breakfast”, but it was still good (seeing there were chocolate muffins and hot coco, my son was quite happy either way!).

The event host, Nigel Poulton, was a great choice by Civo, witty and fun, pleasant to listen to, perfect fit.
After a small introduction, the keynotes was on, a duo consisting of Nick Caldwell and Marty Weiner took the stage.

The keynote was a lot about their experiences surrounding management and their work at reddit, a really giving and relatable talk.
It was quite humoristic, something I enjoy, and even though it was about a hour long, it went by way too fast!

I quite early noted that my son, who only know a little English, found the talks quite… boring (although he totally understood any Smash Bros references) and made the mistake of not taking enough breaks between them during the first day.
What he really seemed to enjoy was the booth area though, and especially the Defence.com table, where he was allowed to learn how to pick locks, something I think is a great thing for a 10-year-old boy to know… ;)

There were a lot of talks and workshops I would have loved to attend, but as always at a conference, you have to pick out the ones you really want to see that are not overlapping. Seeing I brought a 10 y/o with me, that kinda made it obvious that I would have to do other things than just watch talks as well.
I was able to attend quite a few though, and they were all great!

Civo is a cloud native (especially kubernetes focused) company, so a lot of the content at the event was focused around that but not only that, I listened to a Sustainability Panel which was very interesting, went to a WASM workshop, a talk about databases in the cloud and even listened a bit to a talk about AI/ML.
AI and ML was quite prominent at the conference, which is not too weird seeing Civo announced their GPU clusters and improvements to their ML platform at the event, but most of that goes over my head (yes… yes… I really need to read up a bit on it and try it out, I know it’s the coolest thing ever…).

The team was quite busy, but I had the chance to a few short talks with a couple of people from the team - which I really enjoyed, seeing we have only seen each-other through Zoom calls before! - as well as a couple of other of the ambassadors, and of course a lot of the people in the booths!
Very enjoyable, although I’m so unused to all the social stuff that I’m sure I seemed quite awkward, hehe.

At the end of the day, there was a party, which we attended for a bit, but my son was quite tired, so we headed back to the hotel relatively early (after eating at Itsu again of course) and watched the end of the movie we started the day before, and then slept.

Day two of the conference

We got up early the next day as well and headed to the venue right after breakfast, the keynotes the second day was with Kelsey Hightower, a quite well known person in the cloud native world, which I really wanted to listen to.
Just as the keynotes the day before, it was awesome, a lot of the talk was focused on hes life after google (and being retired), after the initial talk he was accompanied by Mark Boost and Dinesh Majrekar from Civo for a discussion and then answering questions from the audience.
I didn’t have the chance to say hello and speak to Kelsey, but from what I saw, he seems to be a very humble person, and even stayed at the event for most of the day just talking to people.

This day, I decided that I would make sure that my son was a bit more stimulated, so between every talk or workshop we attended, we took a stroll in a new direction in Islington.
It was my son - and my - first time in London, so seeing the town was an experience!
We found a nonconformist graveyard (Bunhill Fields) which I had no idea was located there, I even stumbled on the gravestone of William Blake and Daniel Defoe, which made me quite excited.
We visited a few smaller parks and squares and saw quite a bit of Islington, which was fun.

I had the time to watch quite a few talks during day two, notably a panel about Open Source (with Amanda Brock, Peter Zaitsev, Liz Rice, and Matt Barker) which was really giving.
We visited the booth area quite a few times as well, both for me to get the time to speak to the people there, but especially so that my son could keep on working on he’s lock-picking skills!
The event ended with a final talk by the Civo Team and Nigel Poulton and a last visit to the booth area, where my son won a Lego set from Okteto and was gifted a stuffed Kubefirst mascot! (I kinda think that was the best part of the visit for him, possibly challenged by the lock-picks!).

My son wanted to go to Itsu a third time, but I decided that we would actually go a bit further and look for a real restaurant.
We finally decided on hamburgers at a cozy place called “Fat Hippo”, the burgers where really good, but so large that neither of us where able to actually finnish them of.
After eating, we headed back to the hotel and the third day in London was at an end.

Heading home

I won’t dwell too much on this part, but the last day we spent in london was mainly by watching all the “you must see that thing” things in London.
We watched Big Ben, the Palace and a bunch of other things, quite fun and especially rewarding for my son who haven’t been in a city larger than Gothenburg before (The population of London is quite close to the population of our whole country, and 10 times as large as Gothenburg)!

The trip home went flawless (although, me being so nervous made us get to Gatwick way too early, hehe) and we got home to Sweden quite late on the thursday evening.

Final thoughts

I’m extremely happy that I went to the event, and I think that bringing my son was a great thing.
I have missed going to conferences, and Civo Navigate EU as my first in such a long time was probably a perfect match.
I would strongly recommend visiting the next Civo Navigate if you are close to the event, well worth the low ticket price for such a great event. (And I don’t just say that as an ambassador, I really mean it).

Hope to see you at the next Civo Navigate EU!

Cfssl is a great tool for setting up a basic certificate authority. When I wrote my first post about it, I was fairly new to the concept of SSL/TLS and certificates, I researched cfssl to find an easy way to generate certificates for a kubernetes cluster I was working with.
Since then, I’ve tried a few different tools which does similar things, Smallstep CA and Hashicorp Vault are two tools that I use in one or another way, Smallstep CA is great as a server and Vault is a monolith, so, when it comes to locally generating certificates, I still find that I fall back to using cfssl.
So, why would I write another post about cfssl? Well, it’s been over 3 years since I wrote my last one, and it seems to still gather quite a lot of traffic, so I think that it could be useful both for me and for potential readers to get a new, up-to-date post about the tool!

The CFSSL version used in this post is v1.6.3

What is CFSSL?

CFSSL is a toolkit built by Cloudflare, released in 2014. It’s intended to be used to easily create, sign and serve TLS certificates from a small application which can be ran both locally and as a server (rest-ish json api).
The program is written in Go, which makes it easy to build yourself, or just download from GitHub for most OS:es and architectures.
I personally prefer to run it as a docker container and use one of my own creations (shameless plug!).

The program have been used by cloudflare to generate their certificate chains, so from a “is it tested?” perspective, it feels quite sturdy.

How does TLS certificates work?

Each TLS certificate consists of a public and a private certificate. The certificates can be “signed” by an authority, which makes computers and other devices able to identify who issued the certificate and trust them (if they trust the root).
Generating a certificate without getting it signed makes it as much a certificate as if its signed, but each device which wants to trust it will have to add it to their internal trust store.
So, the best way is to have a root certificate, which is made to sign other certificates, which can be added as a trusted root hence all certificates signed by it will be as well.

When we build a certificate “chain”, it’s usually a good idea to create the root on a device which have no access to the internet, the device can then be destroyed (after creating an offline backup of the root and a bunch of child-certs) to keep the root as safe as possible.

There are hardware devices (HSM / Hardware security modules) which can be used to create a certificate more securely, but they are quite pricey and using a raspberry pi-zero or similar would probably be a lot cheaper if you intend to destroy it or stove away the raspberry after generating the certificate.

Each intermediate certificate will be able to create certificates as well, it’s sometimes even worth generating intermediates from the first intermediates, to create a bigger chain and allowing you to easier rotate the certificates further down the chain if needed.
The certificate at the end of the chain is usually called a “leaf” certificate.

Certificate icons created by Smashicons - Flaticon

Installation

To even start using CFSSL we aught to install it. There are (as said earlier) multiple ways to install it, while, if you have go installed (whichever OS you use), you can get the latest version with a simple

# Newer go versions
go install github.com/cloudflare/cfssl/cmd/...@latest

# Older go versions
go get github.com/cloudflare/cfssl/cmd/...

That command will install all the tools included in cfssl, which might not be needed for your case.

Generating the root

If you are just testing the commands and want to see what happens, its totally okay to do it on your local computer, but if you intend to use the root certificate - that you are about to create - for more critical things, be sure to do it on a computer which is offline and won’t be connected to the network again. A production root certificate should be secure, and having it on a machine exposed to the net is not a good idea. If your root certificate runs off on the internet, you will have a HUGE headache and a lot of work to do to rotate all your certs!

To generate a new certificate with CFSSL we need to create a json file with the data that we want the certificate to have.

root.json

{
    "CN": "Jitesoft CA",
    "key": {
        "algo": "ecdsa",
        "size": 384
    },
    "CA": {
        "expiry": "87660h",
        "pathlen": 2
    },
    "names": [
        {
               "C": "SE",
               "L": "Lund",
               "O": "Jitesoft",
               "ST": "Skania"
        }
    ]
}

The above json includes the required data for a ECDSA root certificate for the Jitesoft CA.

The CN property defines the certificates “common name”, the name of the “root”.
Depending on the usage, the CN should have different names, but in my case, I want my top-most certificate to be called my company name and CA to make it known that it’s my certificate authority certificate.

The CA clause allows us to define the pathlen for the certificate as well as the expiry lifetime. Default expiry for cfssl CA’s is 5 years, which might be enough, the example above uses 10 years though.
The pathlen variable indicates how many intermediate certificates that can be created in a hierachy below, 0 means that the CA can only sign the leaf certificates, 1 level of intermediates can be created, 2 that the intermediate certificates can create sub intermediates and so on.

In a certificate used for a webserver, you would set the primary domain as the CN, while you would add a hosts property (an array) with any alternative names (SAN) to make sure that the certificate is bound to the specific domains only. But in the case of a CA, we rather want a generic name than a domain.

The key property defines what type of key it is we want to generate, in this case, I have decided that my certificate should be a ECDSA key with the size of 384 bits.

The final property, names (subject names) gives anyone viewing the certificate a hint of the owner of the certificate.
C = Country (ISO 3166-2 code), L= Locality, O = Organization and ST = state.
If wanted, you may also include OU (organizational unit name), as well as E (email).

RSA or ECDSA

RSA and ECDSA are two algorithms which are quite commonly used for certificates, RSA is quite a lot older and well tested, while ECDSA generates a lot smaller files and makes use of something called “Elliptic Curve Cryptography” (ECC).
ECDSA (or rather Elliptic Curve Digital Signature Algorithm) is a lot more complex than RSA (which instead of the curve makes use of prime numbers).

One issue with choosing an ECC algorithm is that there are software that does not “yet” (after 15+ years) support ECC algorithms. So you should choose an algorithm which is best suited for you to use and especially a size on the root which makes it secure enough (I would recommend using 2048 (or rather higher) with RSA and 384 and over with ECDSA).

The lowest RSA size which CFSSL will accept is 2048 and the highest is 8192, while it accepts 256, 384 as well as 512 while using an elliptic curve algorithm.
As of writing, the RSA and ECDSA algorithms are the only ones supported.

Create the certificate

To create the certificate from the json data we created we invoke the

cfssl gencert command.

cfssl gencert -initca root.json

Now, doing this will create a few values and output it to STDOUT in a json format, but by using the cfssljson tool we can parse it out into a set of files instead:

cfssl gencert -initca root.json | cfssljson -bare root

The gencert command tells cfssl that we want to generate a new certificate (keys and sign request) and by using the -initca option we also tell it that the certificate will be used for a certificate authority.

If you run the ls command you should now find the following new files in the directory: root-key.pem, root.csr and root.pem.

The root.pem is your public key, this can be shared and uploaded everywhere as it’s not a secret (rather the other way around), for a client to validate your signed messages, the certificate needs to be known, and this is done by “trusting” the public key.

The root.csr will not be used with the root certificate, as in this example, we don’t use another CA to sign our certificate.

The root-key.pem is a lot more critical that it does not slip out of your hands. This is the key which will be used to “prove” that your CA is the CA actually signing the other certificates.
It will be used to generate the intermediate certificates and then hidden away.

Verify certificate

With the help of openssl we can quickly verify our new certificate to make sure everything is correct:

> openssl x509 -in root.pem -noout -text
# Prints something like:
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            61:c9:5c:9b:c2:28:32:41:3f:83:7d:ea:b8:82:65:0a:a3:ce:32:32
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft CA
        Validity
            Not Before: Jan  9 13:09:00 2023 GMT
            Not After : Jan  9 23:09:00 2028 GMT
        Subject: C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft CA
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (384 bit)
                pub:
                    04:bc:18:70:4e:18:17:eb:4e:82:6e:b6:8f:83:e3:
                    c8:f3:85:27:a4:20:8f:d2:76:4e:38:9e:7b:6c:5f:
                    4f:ef:60:f8:f8:d1:52:a8:b8:b2:f7:a4:94:fa:f0:
                    cc:f9:c4:45:83:d5:52:29:4b:97:75:72:f3:a2:33:
                    ee:d8:e3:84:ae:bd:1b:a1:9a:54:71:9e:6e:1e:cc:
                    3c:83:ad:1d:78:c2:b5:9b:fb:69:52:ec:5c:79:24:
                    fd:48:9c:39:45:9c:22
                ASN1 OID: secp384r1
                NIST CURVE: P-384
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:2
            X509v3 Subject Key Identifier:
                72:28:B0:15:F5:62:F9:1D:17:CB:03:40:BB:B7:B8:AD:AA:A3:A4:A7
    Signature Algorithm: ecdsa-with-SHA384
         30:65:02:30:01:dd:5e:42:3e:fb:ef:cc:02:2c:ab:96:2d:06:
         ee:95:fc:c7:22:ba:08:db:5d:b6:57:ba:95:0b:52:64:f7:37:
         a5:c1:17:be:ee:ff:0a:87:35:0b:74:4d:1a:69:f6:21:02:31:
         00:83:3d:01:67:d8:c1:f1:96:96:73:cf:00:6d:b3:60:b2:bf:
         2d:05:e0:2e:ee:f7:09:40:41:c8:71:00:cc:b9:ff:31:d5:3e:
         92:39:11:02:8d:1f:a2:37:a1:09:5f:8e:4e

As you can see, the public key shows the client the allowed functionality of the certificate (X509v3 extensions) as well as the information we supplied in the json file before.

Intermediates

When we have our root certificate we will want to create the intermediate certificates which we will later use to sign our leaf certificates with.

To keep our file structure a bit easier to handle as well as easier to display in a blog, create a subfolder for each new intermediate to create.
In my case, I’ll create two: Jitesoft Intermediate 1 and Jitesoft Intermediate 2 and call the folders inter1and inter2 to keep it simple.

mkdir inter1 inter2

CFSSL makes use of a profile concept for generation of new sub-certificates. The profiles configuration can be used for all kinds of certificates, while right now, we just create the intermediate profile:

profiles.json

{
    "signing": {
        "profiles": {
            "intermediate": {
                "usages": [
                    "cert sign",
                    "crl sign"
                ],
                "ca_constraint": {
                    "is_ca": true
                },
                "expiry": "43800h"
            }
        }
    }
}

Each profile requires a set of usages, you can as well define an expiry here (which will replace the value set in the config.json file), and for a Intermediate CA a ca_constraints clause where we set is_ca to true to indicate that it is actually a certificate authority (which an intermediate certificate is).

For an intermediate authority, we need to se the usages cert sign, crl sign.
The cert sign usage allows the CA to sign certificates and the crl sign will allow the certificate to sign certificate revocations.

The profiles file is used when signing the certificates, and just as with the original ca file, we need a configuration file for the specific intermediates:

inter1/config.json

{
    "CN": "Jitesoft Intermediate 1",
    "key": {
        "algo": "ecdsa",
        "size": 384
    },
    "CA": {
        "expiry": "43800h",
        "pathlen": 1
    },
    "names": [
        {
               "C": "SE",
               "L": "Lund",
               "O": "Jitesoft",
               "ST": "Skania"
        }
    ]
}

With those two files, we can generate the intermediate certificate:

cd inter1
cfssl genkey -initca ./config.json | cfssljson -bare inter1

Inspecting the new intermediate certificate will show an unsigned certificate (which is basically the same as the CA) for a client to recognize that it’s issued by your CA, we need to sign it:

cfssl sign -ca ../root.pem -ca-key ../root-key.pem -profile intermediate --config ../profiles.json inter1.csr | cfssljson -bare inter1

In this case, we make use of the csr file (certificate signing request), because we are actually requesting our certificate authority to sign the certificate!

If you re-inspect the certificate with openssl, you will now see that the Issuer have switched from the cert itself (Subject) to the Subject of the CN:

> openssl x509 -in inter1.pem -noout -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            75:74:5e:57:04:c1:06:14:bb:bf:90:3c:93:20:36:bc:0f:38:3a:0d
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft CA
        Validity
            Not Before: Jan  9 13:27:00 2023 GMT
            Not After : Jan  9 14:27:00 2023 GMT
        Subject: C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft Intermediate 1
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (384 bit)
                pub:
                    04:81:4d:6e:ea:b7:0b:c2:b0:80:06:3e:1b:22:9a:
                    84:6f:bc:aa:b5:24:bf:1d:83:4f:70:6f:12:bd:8e:
                    b0:27:cb:e5:7d:a7:8d:f6:da:d3:7d:9e:39:b0:95:
                    07:ae:fa:ad:58:33:72:d5:28:3b:e9:e0:b5:cb:1b:
                    82:2c:30:fa:ce:a7:ab:02:db:1b:a9:1e:15:c8:5a:
                    f8:cc:d2:c8:29:19:07:df:21:89:c6:60:56:b5:bc:
                    08:82:9a:b9:74:ab:5b
                ASN1 OID: secp384r1
                NIST CURVE: P-384
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign, CRL Sign
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                4A:25:E3:2D:62:83:BA:FF:37:3D:C4:A9:B7:13:00:3A:B4:8D:96:C5
            X509v3 Authority Key Identifier:
                keyid:72:28:B0:15:F5:62:F9:1D:17:CB:03:40:BB:B7:B8:AD:AA:A3:A4:A7
    Signature Algorithm: ecdsa-with-SHA384
         30:64:02:2f:0b:e0:46:e4:af:9f:86:23:35:dd:30:79:cd:af:
         91:81:42:b7:cd:c7:90:d8:16:59:0d:43:b7:59:98:cc:65:6f:
         45:17:74:b2:d9:ca:ef:c6:c8:1b:5e:51:62:fd:6d:02:31:00:
         d5:d2:8c:50:be:37:00:15:31:d2:50:84:29:05:cc:d7:4b:17:
         ef:49:8c:d1:6c:a3:5f:06:d1:b7:7d:b9:09:5b:f3:43:46:3e:
         f4:11:16:80:c1:6a:10:8d:af:5a:91:e0

The same can be done with the inter2 to generate a second intermediate!

Leaf!

The whole reason to have a CA is of course to generate certificates, not just new CA:s, and those certificates are the leaves.
Just as with the intermediate profile, the leaf certificate needs a profile with the usages that it requires.

So, we can start with creating two types of certificates, one for server auth and one for client auth:

profiles.json

{
    "signing": {
        "profiles": {
            "intermediate": {
                "usages": [
                    "cert sign",
                    "crl sign"
                ],
                "ca_constraint": {
                    "is_ca": true
                },
                "expiry": "43800h"
            },
            "server": {
                "usages": [
                    "server auth"
                ],
                "expiry": "720h"
            },
            "client": {
                "usages": [
                    "client auth"
                ],
                "expiry": "720h"
            }
        }
    }
}

The two new clauses added are the profiles client and server.

In this example, I’ll create a new directory in the inter1 directory to keep the certificate hierarchy and folder structure as is:

mkdir inter1/certs
cd inter1/certs

We also need to create a configuration for the certificates:

inter1/certs/server.json

{
    "CN": "Server",
    "hosts": [
        "127.0.0.1",
        "server.domain",
        "sub.domain.tld"
    ]
}

inter1/certs/client.json

{
    "CN": "Client",
    "hosts": [""]
}

As you see in the two configurations, we set a CN (common name), which - if this was a web certificate - would contain the primary domain of the page the certificate should be used for, and we add a hosts array, which contains a list of the IP-addresses that the certificate will actually be allowed for.

In this case, the certificates will be used for authentication, so the server have the addresses that it will be served on, while the client have an empty list, as we don’t want the certificate to be only used on one host.

To generate the certificates we - again - use the cfssl tool, but in this case without the -initca flag:

cfssl gencert -ca=../inter1.pem -ca-key=../inter1-key.pem \
  -config=../../profiles.json \
  -profile=server server.json | cfssljson -bare server
  
cfssl gencert -ca=../inter1.pem -ca-key=../inter1-key.pem \
  -config=../../profiles.json \
  -profile=client client.json | cfssljson -bare client

We can now inspect the certificates and see that they are signed with the correct certificate authority (Jitesoft Intermediate 1) that the usages are correct and that the SAN:s are correct:

> openssl x509 -in server.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            07:f9:b7:85:4f:12:a8:10:5c:16:dd:a0:b8:53:80:3c:3c:a4:97:e4
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft Intermediate 1
        Validity
            Not Before: Jan  9 14:10:00 2023 GMT
            Not After : Feb  8 14:10:00 2023 GMT
        Subject: CN = Server
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:52:b6:af:a7:db:dd:0d:2b:0f:ab:d6:49:c7:0e:
                    a8:eb:ef:29:ec:e4:b6:c1:cd:d3:0f:21:f4:5d:a3:
                    b0:ba:c9:b3:11:67:72:20:a7:ec:60:03:76:ec:b0:
                    08:30:14:6e:13:c5:52:66:2b:ec:d2:28:5d:cb:64:
                    a4:06:d9:af:e4
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                AA:83:51:38:F4:95:99:79:AB:3F:77:38:AF:77:CC:37:4A:C2:48:82
            X509v3 Authority Key Identifier:
                keyid:8C:D6:B7:3D:9E:0B:9B:5E:68:82:58:EC:91:84:27:89:FB:58:1B:6E
            X509v3 Subject Alternative Name:
                DNS:server.domain, DNS:sub.domain.tld, IP Address:127.0.0.1
    Signature Algorithm: ecdsa-with-SHA384
         30:65:02:30:7f:c4:45:f2:89:75:5d:ba:ec:32:1a:c8:bd:0a:
         78:c5:c3:fa:86:d3:b9:cf:8d:6f:68:54:54:a1:23:5c:73:7d:
         28:41:11:54:61:55:81:bb:03:5f:f0:be:c7:6a:d5:56:02:31:
         00:bd:16:36:5e:2b:f5:1f:31:25:3c:00:bf:7d:86:fc:eb:91:
         09:ae:05:23:31:8e:51:71:81:da:4b:14:1d:b2:95:16:25:8f:
         9f:49:e8:b4:df:c5:08:dc:e9:d6:5d:cf:58

These certificates had a expiry of 720 hours, so they will only be working for a month. This can ofcourse be changed in the profiles.json file if you want longer certificates!

We can test the chain with openssl and cURL:

# In the `root` directory:
openssl s_server -cert ./inter1/certs/server.pem -key ./inter1/certs/server-key.pem -WWW -port 12345 -CAfile root.pem -verify_return_error -Verify 1
# Open a separate shell and enter the `root` directory:
curl -k --cert ./inter1/certs/client.pem --key ./inter1/certs/client-key.pem https://localhost:12345/test.txt

verify error:num=20:unable to get local issuer certificate

Oh now! This is not good!

This is because openssl (or any other server) can’t verify that the intermediate certificate is actually originating from the root CA. We actually need to bundle the certificates first.

This is done with one of the other tools which is supplied with cfssl, it’s called mkbundle

# in root directory:
mkbundle -f bundle.crt root.pem inter1/inter1.pem

We can cat our new bunlde to see the certificate bundle:

-----BEGIN CERTIFICATE-----
MIICMDCCAbagAwIBAgIUYclcm8IoMkE/g33quIJlCqPOMjIwCgYIKoZIzj0EAwMw
VjELMAkGA1UEBhMCU0UxDzANBgNVBAgTBlNrYW5pYTENMAsGA1UEBxMETHVuZDER
MA8GA1UEChMISml0ZXNvZnQxFDASBgNVBAMTC0ppdGVzb2Z0IENBMB4XDTIzMDEw
OTEzMDkwMFoXDTIzMDEwOTIzMDkwMFowVjELMAkGA1UEBhMCU0UxDzANBgNVBAgT
BlNrYW5pYTENMAsGA1UEBxMETHVuZDERMA8GA1UEChMISml0ZXNvZnQxFDASBgNV
BAMTC0ppdGVzb2Z0IENBMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEvBhwThgX606C
braPg+PI84UnpCCP0nZOOJ57bF9P72D4+NFSqLiy96SU+vDM+cRFg9VSKUuXdXLz
ojPu2OOErr0boZpUcZ5uHsw8g60deMK1m/tpUuxceST9SJw5RZwio0UwQzAOBgNV
HQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB/wIBAjAdBgNVHQ4EFgQUciiwFfVi
+R0XywNAu7e4raqjpKcwCgYIKoZIzj0EAwMDaAAwZQIwAd1eQj7778wCLKuWLQbu
lfzHIroI2122V7qVC1Jk9zelwRe+7v8KhzULdE0aafYhAjEAgz0BZ9jB8ZaWc88A
bbNgsr8tBeAu7vcJQEHIcQDMuf8x1T6SORECjR+iN6EJX45O
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIICWzCCAeCgAwIBAgIUTl02XWipAdn3Y8chGqzegLmrkhAwCgYIKoZIzj0EAwMw
VjELMAkGA1UEBhMCU0UxDzANBgNVBAgTBlNrYW5pYTENMAsGA1UEBxMETHVuZDER
MA8GA1UEChMISml0ZXNvZnQxFDASBgNVBAMTC0ppdGVzb2Z0IENBMB4XDTIzMDEw
OTEzNTYwMFoXDTI4MDEwODEzNTYwMFowYjELMAkGA1UEBhMCU0UxDzANBgNVBAgT
BlNrYW5pYTENMAsGA1UEBxMETHVuZDERMA8GA1UEChMISml0ZXNvZnQxIDAeBgNV
BAMTF0ppdGVzb2Z0IEludGVybWVkaWF0ZSAxMHYwEAYHKoZIzj0CAQYFK4EEACID
YgAEc9LuhhgVEa/Z1CXbYyshJPWjjHNGq8Q88rvU+inxfHCUr/5l10SvwIEaNHiD
FalwWmf/dEtfboPGfI2IaYZZ4A4S8CILK8q90JzpZkPZKpRrdwCSR8BLN3Q7YPVv
Hry0o2MwYTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4E
FgQUjNa3PZ4Lm15ogljskYQniftYG24wHwYDVR0jBBgwFoAUciiwFfVi+R0XywNA
u7e4raqjpKcwCgYIKoZIzj0EAwMDaQAwZgIxAPjNCQcRzPsAPudk0PM7I++B/ihk
kqBaVcVtl75Ru0qCr3T85QEZpQQd6xMLAhOe/QIxAPTwercxV4RwPusrvlLHAqI+
bu3IiUngL2bdz+vU1Pk2i8uzi9kWmL8KocVt+sKXWg==
-----END CERTIFICATE-----

This bundle is the CA certificate that is added to any program which requires the CA.
Now, we just need to modify the openssl command to use the ca bundle instead of the root ca:

openssl s_server -cert ./inter1/certs/server.pem -key ./inter1/certs/server-key.pem -WWW -port 12345 -CAfile bundle.crt -verify_return_error -Verify 1
# Open a separate shell and enter the `root` directory:
curl -k --cert ./inter1/certs/client.pem --key ./inter1/certs/client-key.pem https://localhost:12345/test.txt

And we will have a successful response:

depth=2 C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft CA
verify return:1
depth=1 C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft Intermediate 1
verify return:1
depth=0 CN = Client
verify return:1
FILE:file.txt

And that my friend, is how you set up your own certificate authority and chain.

As always, if you find any issues with the tutorial just let me know, and I’ll update it as soon as possible!

I’ve actually created a bunch of drafts, but none have been published yet as they haven’t been finished…
So.. why? Whats up?

About 1½ years ago I was rushed to the ER. My doctor had called me to report on some tests I had done for my WED, she was a bit startled as the tests had shown that I had acute anemia.
This was kind of a shock, but at the least it explained why I was so tired and almost fained from the smallest tasks (such as walking up the stairs). My blood value was at around 70, which is around 90 units below my standard and I had to be filled up with two bags of blood before they sent me home.

It took a while for me to recover and I had to medicate for a while, but my blood value went up again and the doctors could find no reason for why I was sick.
I went on scheduled tests (weekly, then bi-weekly and then every month) and after a while it returned from nowhere.
All tests re-started and nothing could be found.

As of now, I’m alright, my hB value is okay (over 150 again), my iron levels are still very low and I eat more iron than most people have in their kitchen, but I’m okay.
The ordeal did though take quite a lot of my mental stamina (especially as we still don’t know why I’ve been sick and I still have to go through a lot of testing and examinations).

My company is a single person company, so when I’m sick, there is no income, so most of the time when I’ve not been sick and in bed has been spent on working (and family of course). It goes well, a lot better than I had imagined due to all my problems, but it has forced me to put side-projects (such as this blog and a lot of my open-source stuff) to the side.

But! This is a new year. And a new year obviously means that stuff should be different (right?!), so in the spirit of that, I thought I’d try to get going with the blogging again, it might take some time, as I’m a very slow blogger, but I do hope to have a few new articles out soon.

I’ve been working on a new CFSSL blog post, something I have had requests on re-visiting and I still see a lot of hits on, further I thought I’d try to deep-dive in gRPC and micro-service architecture, something that interests me and I have been researching for a while.

And I guess that’s it, now you know!

As a vivid user of PGP and other signature solutions, cosign is something I’ve been looking at for a while. I just recently started using it to sign container images to be able to implement a bit more strict procedure in my container building and usage flow, something that was a lot easier than I thought!

So, I think that part covers at the least the ‘What’ bit of the excerpt, so let’s head on to ‘Why’.

Signatures and Validation

Most people working in tech have encountered signatures from time to time, it’s often a hazzle if you wish to verify stuff and to keep your own keys can be a real pain. The sigstore project have a few solutions to ease the process (fulcio), but this part will focus more on the signing and verifying bits.

When it comes to why someone would like to sign their images (or binaries or whatever they feel like signing), the reason is quite simple, it’s to allow for consumers to validate the resource and (as long as you are someone they trust) know that what they download, run or install is actually from you and is safe to use.

With my company, I try to make sure that we are both transparent and a source of trustworthy software, and when we release things, I want people to be able to verify that it’s actually from us, not from someone using the name or even hijacking one of our accounts.
Signing and verification consists of two keys and a checksum, the private key is owned by the one who produces the resource, while the public key is to allow validation. The checksum that is the actual signature only fits the private key and if it’s signed with the wrong key, the public key will not accept it as a valid signature.

That way, the trust is in that the key is safe and not used by anyone but the developer while any binary uploaded anywhere can be validated to at the least that point.

There are occurrences when a key runs aloft and malicious software is published with a valid signature, but it’s a lot rarer than seeing an access token or similar get lost and used.

Keeping a “root” key secret is not always easy and depending on the risk of a lost key, there are a lot of different approaches one could take.

I would personally always recommend that one create their root key on a non-online machine and that the machine is wiped after generating the key and some intermediates, but when it comes to this tutorial, we will just use a fresh key and think about the paranoia later!

How

The first thing one have to do to be able to sign their images with cosign is to get ahold of the software.
Cosign can either be installed or ran through docker (or another container runtime). I will describe the ‘install’ way here rather than using docker.

Installation

If you are a go user (which cosign, just as about every new non-frontend-web project now ‘a day is!) and have go 1.6+ installed, you can install the latest version of cosign with a simple go install command:

go install github.com/sigstore/cosign/cmd/cosign@latest

But if you prefer to use binaries, the easiest way is to download the binary from GitHub:

(Linux-ish way!)

# Depending on your architecture you might want to use something other than amd64
ARCH=amd64
LATEST=$(wget -qO- https://api.github.com/repos/sigstore/cosign/releases | jq -r ".[0].name")
wget https://github.com/sigstore/cosign/releases/download/${VERSION}/cosign-linux-${ARCH}
mv cosign-linux-${ARCH} /usr/bin/cosign
chmod +x /usr/bin/cosign

For Windows user, there are exe releases on the cosign release page on GitHub: https://github.com/sigstore/cosign/releases

Test to make sure that it’s installed correctly

cosign version
GitVersion:    v1.2.1
GitCommit:     unknown
GitTreeState:  unknown
BuildDate:     unknown
GoVersion:     go1.16.6
Compiler:      gc
Platform:      linux/amd64

And initialize cosign (creates a .sigstore config directory in your home dir): cosign init.

Create keys!

Now when we have cosign installed we can actually create our first key-pair.
This is simply done by invoking the cosign binary like this:

cosign generate-key-pair

Which produces a cosign.key and a cosign.pub file in the directory where you ran it.
The private key is supposed to be private, really private. If you lose your private key and someone gets a hold of it, they can basically upload binaries with valid signatures in your name, something you really don’t want!

With that said, we can’t throw the key on a usb stick and forget about it for the next 100 years, no, we actually need the key to sign our images!

The public key is for your users. You can distribute it basically however you want, it’s public and totally okay to just throw into a gist or to upload on a warez site! No one can do much with it more than validating your payloads anyway!

So, when we got our two keys, we can basically start sign our images right away.

To sign an image, you use the cosign sign -key cosign.key my-org/my-image, this will create a new signature and upload it to the registry.
To test if it’s a valid image, you use the cosign verify -key cosign.pub my-org/my-image and you should get an output which looks something like this:

Verification for index.docker.io/jitesoft/ubuntu:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
  - Any certificates were verified against the Fulcio roots.

[{"critical":{"identity":{"docker-reference":"index.docker.io/jitesoft/ubuntu"},"image":{"docker-manifest-digest":"sha256:e2700dee042c018ed9505940f6ead1de72155c023c8130ad18cd971c6bfd4f03"},"type":"cosign container image signature"},"optional":{"sig":"jitesoft-bot"}}]

With a lill bit of JQ, you can get it pretty printed as well!:

Verification for index.docker.io/jitesoft/ubuntu:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
  - Any certificates were verified against the Fulcio roots.
[
  {
    "critical": {
      "identity": {
        "docker-reference": "index.docker.io/jitesoft/ubuntu"
      },
      "image": {
        "docker-manifest-digest": "sha256:e2700dee042c018ed9505940f6ead1de72155c023c8130ad18cd971c6bfd4f03"
      },
      "type": "cosign container image signature"
    },
    "optional": {
      "sig": "jitesoft-bot"
    }
  }
]

As you can see in the payload above, there is an ‘optional’ object in the JSON in which I have added a ‘sig’ key. That’s called an annotation, with annotations you can add any arbitrary data to the signature layer pushed to the registry.
Adding annotations is done by the -a flag (can be used multiple times) and then a key=value as the flag value.

Automate it

As many others, I don’t manually build my images, I use CI scripts on GitLab. To sign each and every new image published I’d have to spend most of my days signing images, something I do not like to do, so I want to have it as a part of my build script. The sigstore project supplies a cosign action for GitHub, which can be easily used by a single step:

uses: sigstore/cosign-installer@main

I don’t use GitHub for my pipelines though, so I decided to write my own template for GitLab.
The template can be easily extended from the Jitesoft gitlab-ci-template library if wanted, or you could copy it and modify it after your own choice, as most of the stuff I do outside of client work, it’s released under the MIT license.

The following script is the one that I use:

.sign:
  image: registry.gitlab.com/jitesoft/dockerfiles/cosign:latest
  variables:
    COSIGN_ANNOTATIONS: ""
  before_script:
    - if [ -z ${COSIGN_PUB_KEY_PATH+x} ]; then echo "Failed to find public key"; exit 1; fi
    - if [ -z ${COSIGN_PRIV_KEY_PATH+x} ]; then echo "Failed to find private key"; exit 1; fi
    - |
      if [ ! -z ${DOCKER_CRED_FILE+x} ]; then
        mkdir ~/.docker
        cp ${DOCKER_CRED_FILE} ~/.docker/config.json
      fi
    - |
      if [ ! -z ${SIGN_IMAGES+x} ] && [ ! -z ${SIGN_TAGS} ]; then
        wget https://gist.githubusercontent.com/Johannestegner/093e8053eabd795ed84b83e9610aed6b/raw/helper.sh
        chmod +x helper.sh
        COSIGN_TAGS=$(./helper.sh imagelist "${SIGN_IMAGES}" "${SIGN_TAGS}")
      elif [ -z ${COSIGN_TAGS+x} ]; then
        echo "Failed to find tags to sign"
        exit 1
      fi
  script:
    - |
      for IMAGE in ${COSIGN_TAGS};
      do
        cosign sign ${COSIGN_ANNOTATIONS} -key ${COSIGN_PRIV_KEY_PATH} ${IMAGE}
      done

OBSERVE if you run this script, only allow it on protected branches, and if possible, make sure you use your own runners. And as I have said many times in this post… DO NOT LOSE YOUR PRIVATE KEY!

As you can see, I do a bit more than just sign the image, so I’ll briefly explain how it works.

The image used is an image built (and signed of course!) under the jitesoft organisation.
It’s based on alpine linux to allow for a small-ish image but still the ability to run stuff interactively (the official cosign image is a distroless image).
To skip using a docker run in the configuration, I use the image as the actual image the scripts run in.

It verifies that there are two environment variables (which I personally use gitlab secrets for) are set (should point to the path of the public and private key, even though the public key is not currently used) and then, if there is a docker credentials json file, it moves it to the home folder of the non-root user.

To make it even more easy for myself, I decided to include a small helper script in case the SIGN_IMAGES and SIGN_TAGS variables are set, it basically loops through all the images and gives them the same tags on all (as you can see in the helper.sh execution in the script).

When the tags are set up (or there already is a COSIGN_TAGS variable set) the script moves on to actually signing the tags with the cosign sign command. Any optional annotations is included with the COSIGN_ANNOTATIONS variable.

Final words

I hope that this little tutorial gave some insight on why one would want to use cosign and similar tools and also how to use it in a simple way.

I will keep on using cosign and update or create a new post about it in the future, especially when I start testing fulcio in a larger scale!

AArch64 is a part of Fosshost. Fosshost is a non-profit organisation that provides open-source projects with both x86_64 and aarch64 machines. My company, Jitesoft, uses both platforms to host a few of the GitLab runners that build our docker images.

AArch64/ARM64

AArch64 is a common name for the 64-bit version of the ARM architecture. All Jitesoft Docker images are built for both arm64 and x86_64 - and, if possible, others as well.

When we build an image with compiled binaries, we try to use the correct architecture for the builders. By doing that, we don’t have to set up a toolchain, and we don’t have to use Qemu or similar software to emulate the builds. When the binary is built, we mount it with the help of buildkit and copy it over to the image during the image build phase. This way, we can keep the image layers small without having to squash them and allowing us to build for many platforms.

Initially, we built them all during the image creation, with x86_64 machines only, but compiling larger projects using Qemu (which is, or at the least was at the time, single-core) made the compile times span over 10+ hours per architecture for some projects. Ten hours compile time was not feasible, so real Arm-based hardware was required.

The before times

When we started to build binaries for the ARM architecture, on real Arm-based hardware, the CI runners were deployed to machines at Scaleway (back then, they had pretty cheap Arm-based machines), but they ended this offering about a year ago. Linaro accepted us as a tenant on their ARM labs for a while, but they discontinued it a while back as well.

When we lost our runners at Linaro, we had to find something new, which was hard. Arm-based providers are not always cheap, and most of the ones we evaluated were quite a bit over our budget for an open-source project. We didn’t want to stop building the docker images, so a couple of Raspberry Pis were bought to run as dedicated CI servers.

RPi’s are pretty good for their price and size, but they are far from as powerful as a “real” machine, so the compilations were again taking way too much time. And then AArch64.com was launched!

Working with AArch64 (and ipv6)

The machine we run at AArch64.com is a lovely 8 core, 16GB RAM machine. The storage is sparse but, with a simple cronjob to clear the docker images stored on disk every now and then, that’s not an issue. But, we are using an IPv6 only network. I could have requested an IPv4 from Fosshost, but seeing that the machine doesn’t need an IP (more than for ssh), I figured it would not be necessary. Taking up an IPv4 for something that will just run without any access to the external network is not something one should do.

The machines themselves are already set up to use Cloudflares’ IPv6 DNS, so package downloading and such works out of the box! On our build servers, we only install the most necessary software: docker, the buildx docker-cli plugin, gitlab-runner (as GitLab is where we host our code, run our CI and initially publish our images) and any required dependencies. IPTables is set up to allow any outgoing traffic and only accept SSH in.

I’m not used to working with IPv6 at all! I wasn’t even aware of the fact that I didn’t have IPv6 at home. If a machine uses ipv6 and your ISP does not support it, you might have to connect through a so-called “jump-host” (which AArch64.com provides!), which I figured out quickly thanks to the documentation on the platform. Running Docker with IPv6, though… that took a bit more work to figure out!

Some people use Docker and IPv6, but reading through the net made me feel that it’s not something many people do. The steps to take are not extreme: the changes are made to the docker daemon, and some tweaks should be done to the firewall as well. It’s nothing complex, but without prior knowledge, it was a bit tricky!

First of all, the docker daemon has to be updated (create or edit the /etc/docker/daemon.json file)

{
  "ipv6": true,
  "fixed-cidr-v6": "fd5f:a3e1:47c8:c8f4::/64"
}

The ipv6: true flag will enable IPv6 in docker, but we also have to set a CIDR for the docker network to use. The value you add there should be a private ipv6 range; use any you wish, or even the above, as it was randomly generated! The size of 64 might not be needed, but a few IP addresses are available, so it should be quite fine.

Now, by restarting the docker daemon (systemctl restart docker) the default network for docker should be using IPv6! This will enable IPv6 on the internal docker networks and for incoming/outgoing traffic.

Finally, we need to masq the traffic for docker. To do this, we need to add a postrouting rule on the IPv6 table, as docker does not do this itself:

ip6tables -t nat -A POSTROUTING -s fd00::/80 ! -o docker0 -j MASQUERADE

A simple reconfigure of the iptables-persist (or installation) and we are ready to reboot if wanted!

Final words

Our pipelines are set up to only share the docker socket on protected branches, branches only maintainers are allowed to merge and push to, so we allow our docker images to build privileged. This is unsafe if the builders are running on branches that are not protected. If you plan to do such things, make sure you read up on rootless docker and how that works, as it will allow you to configure docker without root privileges. Further, there are other OCI image builders do not require root, such as podman.

Protecting your servers and build environments against potential attacks is extremely important. If you are compromised and you accidentally publish images with malware or security holes, there are potentially thousands of people who might be at risk.

Building open-source is wonderful; the knowledge that people like and use the things you create is excellent. We are very grateful to Fosshost and the AArch64.com project for their contribution of server power to allow us to keep on doing it.