I really like the ease of setting it up, and the fact that they support a wide array of database engines makes their operators very useful.
In this post, we will focus on XtraDB, which is their mysql version with backup and clustering capabilities.
We will go through the installation of the operator as well as what I find most important in the
custom resource which will allow us to provision a full XtraDb cluster with backups and proxying.
This is what I’ve been using the most, and I’ll try to create a post at a later date with some benchmarks to show
how it compares with other databases.
Running databases in kubernetes (or docker) have earlier been a big no-no, this is not as much of an issue now ‘a days, especially
when using good storage types.
In this writeup, I’ll use my default storage class which on my k3s cluster is a mounted disk on Hetzner, they are
decent in speed, but seeing it’s just a demo, the speed doesn’t matter much!
Percona xtradb makes use of cert-manager to generate TLS certificated, it will automatically
create an issuer (namespaced) for your resources, but you do need to have cert-manager installed.
This post will not cover the installation, and I would recommend that you take a look at the
official cert-manager doucmentation for installation instructions.
The first thing we have to do is to install the helm charts and deploy them to a kubernetes cluster.
In this post, we will as I said earlier, use the Mysql version of percona, and we will use the operator that is
supplied by percona.
If you want to dive deeper, you can find the documentation here!
Percona supplies their own helm charts for the operator via GitHub, so adding it to helm is easily done with
helm repo add percona https://percona.github.io/percona-helm-charts/
helm repo update
If you haven’t worked with helm before, the above snippet will add it to your local repository and allow you to install charts from the repo we add.
If you just want to install the operator right away, you can do this by invoking the helm install
command, but we
might want to look a bit at the values we can pass to the operator first, to customize it slightly.
The full chart can be found on GitHub, where you should
be able to see all the customizable values in the values.yml
file (the ones set are the default values).
In the case of this operator, the default values are quite sane, it will create a service account and set up the RBAC
values required for it to monitor the CRD:s.
But, one thing that you might want to change is the value for watchAllNamespaces
.
The default value here is false
, which will only allow you to create new clusters in the same namespace
as the operator. This might be a good idea if you have multiple tenants in the cluster, and you don’t want
all of them to have access to the operator, while for me, making it a cluster-wide operator is far more useful.
To tell the helm chart that we want it to change the said value, we can either pass it directly in the install
command, or we can set up a values
file for our specific installation.
When you change a lot of values, or you want to source-control your overrides, a file is for sure more useful.
To create an override file, you need to create a values.yml
(you can actually name it whatever you want)
where you set the values you want to change, the format is the same as in the above repository values.yml file,
so if we only want to change the namespaces parameter it would look like this:
watchAllNamespaces: true
But any value in the default values file can be changed.
Installing the operator with the said values file is done by invoking the following command:
helm install percona-xtradb-operator percona/pxc-operator -f values.yml --namespace xtradb --create-namespace
The above command will install the chart as percona-xtradb-operator
in the xtradb
namespace.
You can change namespace as you wish, and it will create the namespace for you.
If you don’t want the namespace to be created (using another one or default) skip the --create-namespace
flag.
Without using the namespace flag, the operator will be installed in the default
namespace.
The file we changed is passed via the -f
flag, and will override any values already defined in the default values file.
When we set the watchAllNamespaces
value, the helm installation will create cluster wide roles and bindings, this does not happen if it’s not set
but is required for the operator to be able to look for and manage clusters in all namespaces.
If you don’t want to use a custom values file, passing values to helm is done easily by the following flags:
helm install percona-xtradb-operator percona/pxc-operator --namespace xtradb --set watchAllNamespaces=true
Currently, the operator images (and other as well) are only available for the AMD64 architecture, so in cases where you use nodes
which are based on another architecture (like me who use a lot of ARM64), you might want to set the nodeSelector
value in your override to only use amd64 nodes:
nodeSelector:
kubernetes.io/arch: amd64
To update your installation, instead of using install
(which will make helm yell about already having it installed)
you use the upgrade command:
helm upgrade percona-xtradb-operator percona/pxc-operator -f values.yml --namespace xtradb
If you are lazy like me, you can actually use the above command with the --install
flagg to install as well.
As with most operators, the xtradb operator comes with a few custom resource definitions to allow easy creation of new clusters.
We can install a new db cluster with helm as well, but I prefer to version control my resources and I really enjoy using the CRD:s supplied by operators I use, so we will go with that!
So, to install a new percona xtradb cluster, we will create a new kubernetes resource as a yaml manifest.
The cluster uses the api version pxc/percona.com/v1
and the kind we are after is PerconaXtraDBCluster
.
There are a lot of configuration that can be done, and a lot you really should look deeper into if you are
intending to run the cluster in production (especially the TLS options and how to encrypt the data at rest).
But to keep this post under a million words, I’ll focus on the things we need to just get a cluster up and running!
As with all kubernetes resources, we will need a bit of metadata to allow kubernetes to know where and what to create:
apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
name: cluster1-test
namespace: private
In the above manifest, I’m telling kubernetes that we want a PerconaXtraDBCluster set up in the private
namespace
using the name cluster1-test
.
There are a few extra finalizers we can add to the metadata to hint to the operator how we want it to handle removal of
clusters, the ones that are available are the following:
These might be important to set up correctly, as they will allow for the operator to remove PVC:s and other
configurations which we want it to remove on cluster deletion.
If you do want to save the claims and such, you should not include the finalizers in the metadata.
After the metadata have been set, we want to start working on the specification of the resource.
There is a lot of customization tha can be done in the manifest, but the most important sections are the following:
tls
(which allows us to use cert-manager to configure mTLS for the cluster)upgradeOptions
(which allows us to set up upgrades of the running mysql servers)pxc
(the configuration for the actual percona xtradb cluster)haproxy
(configuration for the HAProxy which runs in front of the cluster)proxysql
(configuration for the ProxySQL instances in front of the cluster)logcollector
(well, for logging of course!)pmm
(Percona monitor and management, which allows us to monitor the instances)backup
(this one you can probably guess the usage for!)In this writeup I will leave this with the default values (and not even add it to the manifest), that way the cluster will create its own issuer and just issue tls certificates as it needs to, but if you want the certificate to be a bit more constrained, you can here set boh which issuer to use (or create) as well as the SANs to use.
Keeping your database instances up to date automatically is quite a sweet feature. Now, we don’t always want to do this
seeing we sometimes want to use the exact same version in the cluster as in another database (if we got multiple environments for example)
or if we want to stay on a version we know is stable.
But, if we want to live on the edge and use the latest version, or stay inside a patch version of the current version we use
this section is very good.
There are three values that can be set in the upgradeOptions
section, and they handle the scheduling, where to look and
the version constraints we want to sue.
upgradeOptions:
versionServiceEndpoint: ' https://check.percona.com'
apply: '8.0-latest'
schedule: '0 4 * * *'
The versionServiceEndpoint
flag should probably always be https://check.percona.com
, but if there are others
you can probably switch. I’m not sure about this though, so to be safe, I keep it at the default one!
apply
can be used to set a constraint or disable the upgrade option all together.
If you don’t want your installations to upgrade, just set it to disabled
, then it will not run at all.
In the above example, I’ve set the version constraint to use the latest
version of the 8.0 mysql branch.
This can be set to a wide array of values, for more detailed info, I recommend checking the percona docs.
The Schedule is a cron-formatted value, in this case, at 4am every day, to continuously check, set it to * * * * *
!
The pxc section of the manifest handles the actual cluster setup.
It got quite a few options, and I’ll only cover the ones I deem most important to just run a cluster, while
as said earlier, if you are intending to run this in production, make sure you check the documentation or read the
CRD specification for all available options.
spec:
pxc:
nodeSelector:
kubernetes.io/arch: amd64
size: 3
image: percona/percona-xtradb-cluster:8.0.32-24.2
autoRecovery: true
expose:
enabled: false
resources:
requests:
memory: 256M
cpu: 100m
limits:
memory: 512M
cpu: 200m
volumeSpec:
persistentVolumeClaim:
storageClassName: 'hcloud-volumes'
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
The size variable will tell the operator how many individual mysql instances we want to run.
3 is a good one, seeing most clustered programs prefer 3 or more instances.
The image should presumably be one of the percona images in this case, to allow updates and everything to work as smoothly as possible.
I haven’t peeked enough into the images, but I do expect that there is some custom things in the images
to make them run fine, which makes me want to stick to the default images rather than swapping to another!
autoRecovery
should probably almost always be set to true
, this will allow the Automatic Crash Recovery
functionality to work, which I expect is something most people prefer to have.
I would expect that you know how resources
works in kubernetes, but I included it in the example to make sure that
its seen, as you usually want to be able to set those yourself. The values set above are probably
quite a bit low when you want to be able to use the database for more than just testing, so set them accordingly,
just remember that it’s for each container, not the whole cluster!
The volumeSpec
is quite important. In the above example, I use my default volume type, which is a RWO type of disk
the size is set to 10Gi. The size should probably either be larger or possible to expand on short notice.
There are two more keys which can be quite useful if you wish to customize your database a bit more, and especially
if you want to finetune it.
Percona xtradb comes with quite sane defaults, but when working with databases, it’s not unusual that you need
to enter some custom params to the my.cnf
file.
The percona pxc configuration does not currently allow bare environment variables (from what I can see), but this is not
a huge issue, seeing the spec allows for a envVarsSecret
to be set.
The secret must of course be in the same namespace as the resources, but any variables in it will be loaded as
environment variables into the pod.
I’m not certain what environment variables are available for the pxc section, but will try to update this part when I got more
info on it.
The configuration
property expects a string, the string is a mysql configuration file, i.e, the values that you usually
put in the my.cnf
file.
spec:
pxc:
configuration: |
[mysqld]
innodb_write_io_threads = 8
innodb_read_io_threads = 8
Percona allows you to choose between two proxies to use for loadbalancing, which is quite nice.
The available proxies are HAProxy and ProxySQL, both
valid choices which are well tried in the industry for loadbalancing and proxying.
The one you choose should have the property enabled
set to true, and the other one set to false.
The most “default” configuration you can use would look like this:
# With haproxy
spec:
haproxy:
nodeSelector:
kubernetes.io/arch: amd64
enabled: true
size: 3
image: percona/percona-xtradb-cluster-operator:1.13.0-haproxy
resources:
requests:
memory: 256M
cpu: 100m
# With proxysql
spec:
proxysql:
nodeSelector:
kubernetes.io/arch: amd64
enabled: true
size: 3
image: percona/percona-xtradb-cluster-operator:1.13.0-proxysql
resources:
requests:
memory: 256M
cpu: 100m
volumeSpec:
emptyDir: {}
The size should be at the least 2 (can be set to 1 if you use allowUnsafeConfigurations
but that’s not recommended).
The image is just as with the pxc configuration most likely best to use the percona provided images (in this case 1.13.0, same version as the percona operator).
As always, the resources aught to be finetuned to fit your needs, the above is on the lower end, but could work okay for a smaller cluster which does not have huge traffic.
Both of the sections allow for (just as with pxc section) to supply both environment variables via the envVarsSecret
as
well as a configuration
property. The configuration does of course differ and I would direct you to the proxy documentation
for more information about those!
Now, something quite important to note here is that if you supply a configuration file, you need to supply the full file,
it doesn’t merge the default file but replaces it in full.
So if you want to finetune the configuration, include the default configuration as well (and change it), this
applies to both haproxy and proxysql and will work the same if you use a configmap, secret or directly accessing
the configuration
key.
The choice of proxy might be important to decide on at creation of the resource, if you use proxysql, you can (with a restart of the pods) switch to haproxy, while if you choose haproxy, you can’t change the cluster to use proxysql. So I would highly recommend that you decide which to use before creating the cluster.
There are a lot more variables you can set here, and all of them can be found at the documentation page.
Logs are nice, we love logs! Percona seems to as well, because they supply us with a section for configuring a fluent bit log collector right in the manifest! No need for any sidecars, just turn it on and start collecting :)
If you already have some type of logging system which captures all pods logs and such, this might not be useful
and you can set the enabled
value to false
and ignore this section.
The log collector specification is quite slim, and looks something like this:
spec:
logcollector:
enabled: true
image: percona/percona-xtradb-cluster-operator:1.13.0-logcollector
resources:
requests:
memory: 64M
cpu: 50m
configuration: ...
The default values might be enough, but the fluent bit documentation got quite a bit of customization available if you really want to!
The xtradb server is able to push metrics and monitoring data to a PMM (percona monitoring & management) service, now, this is not installed with the cluster and needs to be set up separately, but if you want to make use of this (which I recommend, seeing how important monitoring is!) the documentation can be found here.
I haven’t researched this too much yet, but personally I would have loved to be able to scrape the instances with prometheus and have my dashboards in my standard Grafana instance, which I will ask percona about if it’s possible. In either case, I’ll update this part with more information when I figure it out!
Backups, one of the most important parts of keeping a database up and running without angry customers questioning you about where their 5 years of data has gone after a database failure… Well, percona helps us with that too, thankfully!
The percona backup section allows us to define a bunch of different storage engines to use to store our backups, this is great, because we don’t always want to store our backups on the same disks or systems as we run our cluster. The most useful way to store backups is likely to store them in a s3 compatible storage, which can be done, but if you really want to you can store them either in a PV or even on the local disk of the node. We can even define multiple storages to use with different schedules!
spec:
backup:
image: perconalab/percona-xtradb-cluster-operator:main-pxc8.0-backup
storages:
s3Storage:
type: 's3'
nodeSelector:
kubernetes.io/arch: amd64
s3:
bucket: 'my-bucket'
credentialsSecret: 'my-credentials-secret'
endpointUrl: 'the-s3-service-i-like-to-use.com'
region: 'eu-east-1'
local:
type: 'filesystem'
nodeSelector:
kubernetes.io/arch: amd64
volume:
persistentVolumeClaim:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10G
schedule:
- name: 'daily'
schedule: '0 0 * * *'
keep: 3
storageName: s3Storage
- name: 'hourly'
schedule: '0 * * * *'
keep: 2
storageName: 'local'
In the above yaml, we have set up two different storage types. One s3
type and one filesystem
type.
The s3 type is pointed to a bucket in my special s3-compatible storage while the filesystem one makes use of a persistent volume.
In the schedule
section, we set it to create a daily backup to the s3 storage (and keep the 3 latest ones) while the
local storage one will keep 3 and run every hour.
Each section under storages
will spawn a new container, so we can change the resources and such for each of them (and you might want to)
and they will by default re-try creation of the backup 6 times (can be changed by setting the spec.backup.backoffLimit
to a higher value).
There is a lot of options for backups, and I would highly recommend to take a look at the docs for it!
One thing that could be quite useful when working with backups for databases is point in time recovery.
Percona xtradb have this available in the backup section under the pitr
section:
spec:
backup:
pitr:
storageName: 'local'
enabled: true
timeBetweenUploads: 60
It makes use of the same storage
as defined in the storages
section, and you can set the interval on PIT uploads.
Sometimes our databases fails very badly, or we get some bad data injected into it. In cases like those
we need to restore an earlier backup of said database.
I won’t cover this in this blogpost, as it’s too much to cover in a h4 in a tutorial like this,
but I’ll make sure to create a new post with disaster scenarios and how percona handles them.
If you really need to recover your data right now (before my next post) I would recommend that you either read the Backup and restore and “How to restore backup to a new kubernetes-based environment” section in the documentation.
Now, when we have had a look at the different sections, we can set up our full chart:
apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
name: cluster2-test
namespace: private
spec:
upgradeOptions:
versionServiceEndpoint: ' https://check.percona.com'
apply: '8.0-latest'
schedule: '0 4 * * *'
pxc:
size: 3
nodeSelector:
kubernetes.io/arch: amd64
image: percona/percona-xtradb-cluster:8.0.32-24.2
autoRecovery: true
expose:
enabled: false
resources:
requests:
memory: 256M
cpu: 100m
limits:
memory: 512M
cpu: 200m
volumeSpec:
persistentVolumeClaim:
storageClassName: 'hcloud-volumes'
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
haproxy:
enabled: true
nodeSelector:
kubernetes.io/arch: amd64
size: 3
image: percona/percona-xtradb-cluster-operator:1.13.0-haproxy
resources:
requests:
memory: 256M
cpu: 100m
proxysql:
enabled: false
logcollector:
enabled: true
image: percona/percona-xtradb-cluster-operator:1.13.0-logcollector
resources:
requests:
memory: 64M
cpu: 50m
backup:
image: perconalab/percona-xtradb-cluster-operator:main-pxc8.0-backup
storages:
s3Storage:
type: 's3'
nodeSelector:
kubernetes.io/arch: amd64
s3:
bucket: 'my-bucket'
credentialsSecret: 'my-credentials-secret'
endpointUrl: 'the-s3-service-i-like-to-use.com'
region: 'eu-east-1'
local:
type: 'filesystem'
nodeSelector:
kubernetes.io/arch: amd64
volume:
persistentVolumeClaim:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10G
schedule:
- name: 'daily'
schedule: '0 0 * * *'
keep: 3
storageName: s3Storage
- name: 'hourly'
schedule: '0 * * * *'
keep: 2
storageName: 'local'
pitr:
storageName: 'local'
enabled: true
timeBetweenUploads: 60
Now, to get the cluster running, just invoke kubectl and it’s done!
kubectl apply -f my-awesome-cluster.yml
It takes a while for the databases to start up (there are a lot of components to start up!) so you might have to wait a few minutes
before you can start play around with the database.
Check the status of the resources with the get
kubectl command:
kubectl get all -n private
NAME READY STATUS RESTARTS AGE
pod/cluster1-test-pxc-0 3/3 Running 0 79m
pod/cluster1-test-haproxy-0 2/2 Running 0 79m
pod/cluster1-test-haproxy-1 2/2 Running 0 78m
pod/cluster1-test-haproxy-2 2/2 Running 0 77m
pod/cluster1-test-pxc-1 3/3 Running 0 78m
pod/cluster1-test-pxc-2 3/3 Running 0 76m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) 3y120d
service/cluster1-test-pxc ClusterIP None <none> 3306/TCP,33062/TCP,33060/TCP 79m
service/cluster1-test-pxc-unready ClusterIP None <none> 3306/TCP,33062/TCP,33060/TCP 79m
service/cluster1-test-haproxy ClusterIP 10.43.45.157 <none> 3306/TCP,3309/TCP,33062/TCP,33060/TCP 79m
service/cluster1-test-haproxy-replicas ClusterIP 10.43.54.62 <none> 3306/TCP 79m
NAME READY AGE
statefulset.apps/cluster1-test-haproxy 3/3 79m
statefulset.apps/cluster1-test-pxc 3/3 79m
When all StatefulSets are ready, you are ready to go!
When a configuration as the one above is applied, a few services will be created.
The service you most likely want to interact with is called <your-cluster-name>-haproxy
(or -proxysql
depending on proxy)
which will proxy your commands to the different backend mysql servers.
From within the cluster it’s quite easy, just accessing the service, while outside will
require a loadbalancer service (which can be defined in the manifest) alternatively a ingress which can
expose the service to the outer net.
If you wish to test your database from within the cluster, you can run the following command:
kubectl run -i --rm --tty percona-client --namespace private --image=percona:8.0 --restart=Never -- bash -il
percona-client:/$ mysql -h cluster1-haproxy -uroot -proot_password
The root password can be found in the <your-cluster-name>-secrets
secret under the root
key.
I really enjoy using percona xtradb, it allows for really fast setup of mysql clusters with backups enabled
and everything one might need.
But, I’m quite new to the tool, and might have missed something vital or important!
So please, let me know in the comments if something really important is missing or wrong.
As you might know, I’m an ambassador for Civo, a cloud native service provider. Since about a year ago they started to host conferences, where the first one was in the US, and now in september, the first EU one was hosted in London at The Brewery, and I was invited.
I’ve earlier written about my issues with anxiety, something which makes traveling to another country for a couple of
days kinda hard. So to make it easier for me, I decided to bring my 10-year-old son (I was assured that it would be
child-friendly enough before hehe).
I’ll try not to dwell in this kind of issues in this post, seeing it’s quite irrelevant, but I can tell you that it
all went very well (with a tiny issue where I bought a ticket with train to Maregate instead of Moorgate and almost ended up way off course!).
The Brewery was more than I expected! It was a really hot period in London while we were there (above 30c every day, in September!!),
but the ventilation at the brewery was so good that I even had to wear my hoodie during some of the talks!
Over the two days of the conference, food (which I gathered was made at The Brewery) was served. A lot of it was stuff I (nor my son) had not tried before, and it tasted fantastic.
The areas for the talks where of great size and the staff succeeded greatly in both sound and light during all talks we attended.
The rooms used for workshops where large enough while still cozy.
The brewery have quite a history (with brewing of beer if you can imagine), something I would recommend checking out at their site if you are interested in brewing history!
We decided to pre-register the day before the event started, and got to meet some of the Civo Staff, whom where very friendly and easy to talk to, we then took a small stroll through Islington and ended up eating sushi at Itsu, a place which ended up being the only place my son wanted to eat (he really loves sushi).
We headed back to the hotel quite early and watched a movie to be able to get up early for the keynotes.
We succeeded in getting up at a decent time to grab some breakfast at the hotel before heading out.
We were both a tad bit sad that there was no real “English Breakfast”, but it was still good (seeing there were chocolate muffins and hot coco, my son was quite happy either way!).
The event host, Nigel Poulton, was a great choice by Civo, witty and fun, pleasant to listen to, perfect fit.
After a small introduction, the keynotes was on, a duo consisting of Nick Caldwell and Marty Weiner took the stage.
The keynote was a lot about their experiences surrounding management and their work at reddit, a really giving and
relatable talk.
It was quite humoristic, something I enjoy, and even though it was about a hour long, it went by way too fast!
I quite early noted that my son, who only know a little English, found the talks quite… boring (although he totally understood any Smash Bros references)
and made the mistake of not taking enough breaks between them during the first day.
What he really seemed to enjoy was the booth area though, and especially the Defence.com table, where he was allowed to learn how
to pick locks, something I think is a great thing for a 10-year-old boy to know… ;)
There were a lot of talks and workshops I would have loved to attend, but as always at a conference, you have to pick out
the ones you really want to see that are not overlapping. Seeing I brought a 10 y/o with me, that kinda made it obvious that I would
have to do other things than just watch talks as well.
I was able to attend quite a few though, and they were all great!
Civo is a cloud native (especially kubernetes focused) company, so a lot of the content at the event was focused around that
but not only that, I listened to a Sustainability Panel which was very interesting, went to a WASM workshop, a talk about databases in the cloud and even listened a bit to
a talk about AI/ML.
AI and ML was quite prominent at the conference, which is not too weird seeing Civo announced their GPU clusters and improvements to
their ML platform at the event, but most of that goes over my head (yes… yes… I really need to read up a bit on it and try it out, I know it’s the coolest thing ever…).
The team was quite busy, but I had the chance to a few short talks with a couple of people from the team - which I really enjoyed, seeing
we have only seen each-other through Zoom calls before! - as well as a couple of other of the ambassadors, and of course a lot of the people in the booths!
Very enjoyable, although I’m so unused to all the social stuff that I’m sure I seemed quite awkward, hehe.
At the end of the day, there was a party, which we attended for a bit, but my son was quite tired, so we headed back to the hotel relatively early (after eating at Itsu again of course) and watched the end of the movie we started the day before, and then slept.
We got up early the next day as well and headed to the venue right after breakfast, the keynotes the second day was with
Kelsey Hightower, a quite well known person in the cloud native world, which I really wanted to listen to.
Just as the keynotes the day before, it was awesome, a lot of the talk was focused on hes life after google (and being retired), after the
initial talk he was accompanied by Mark Boost and Dinesh Majrekar from Civo for a discussion and then
answering questions from the audience.
I didn’t have the chance to say hello and speak to Kelsey, but from what I saw, he seems to be a very humble person, and
even stayed at the event for most of the day just talking to people.
This day, I decided that I would make sure that my son was a bit more stimulated, so between every talk or workshop we attended,
we took a stroll in a new direction in Islington.
It was my son - and my - first time in London, so seeing the town was an experience!
We found a nonconformist graveyard (Bunhill Fields) which I had no idea was located there, I even stumbled on the gravestone
of William Blake and Daniel Defoe, which made me quite excited.
We visited a few smaller parks and squares and saw quite a bit of Islington, which was fun.
I had the time to watch quite a few talks during day two, notably a panel about Open Source
(with Amanda Brock, Peter Zaitsev, Liz Rice, and Matt Barker) which was really giving.
We visited the booth area quite a few times as well, both for me to get the time to speak to the people there, but especially
so that my son could keep on working on he’s lock-picking skills!
The event ended with a final talk by the Civo Team and Nigel Poulton and a last visit to the booth area, where my son
won a Lego set from Okteto and was gifted a stuffed Kubefirst mascot! (I kinda think that was the best part of the visit for him, possibly challenged by the lock-picks!).
My son wanted to go to Itsu a third time, but I decided that we would actually go a bit further and look for a real restaurant.
We finally decided on hamburgers at a cozy place called “Fat Hippo”, the burgers where really good, but so large that neither of us
where able to actually finnish them of.
After eating, we headed back to the hotel and the third day in London was at an end.
I won’t dwell too much on this part, but the last day we spent in london
was mainly by watching all the “you must see that thing” things in London.
We watched Big Ben, the Palace and a bunch of other things, quite fun and especially rewarding for my son who haven’t been in a city larger than Gothenburg before
(The population of London is quite close to the population of our whole country, and 10 times as large as Gothenburg)!
The trip home went flawless (although, me being so nervous made us get to Gatwick way too early, hehe) and we got home to Sweden quite late on the thursday evening.
I’m extremely happy that I went to the event, and I think that bringing my son was a great thing.
I have missed going to conferences, and Civo Navigate EU as my first in such a long time was probably a perfect match.
I would strongly recommend visiting the next Civo Navigate if you are close to the event, well worth the low ticket price
for such a great event. (And I don’t just say that as an ambassador, I really mean it).
Hope to see you at the next Civo Navigate EU!
]]>Cfssl is a great tool for setting up a basic certificate authority. When I wrote my first post about it, I was fairly
new to the concept of SSL/TLS and certificates, I researched cfssl to find an easy way to generate certificates for
a kubernetes cluster I was working with.
Since then, I’ve tried a few different tools which does similar things, Smallstep CA and Hashicorp Vault are two tools
that I use in one or another way, Smallstep CA is great as a server and Vault is a monolith, so, when it comes to locally
generating certificates, I still find that I fall back to using cfssl.
So, why would I write another post about cfssl? Well, it’s been over 3 years since I wrote my last one, and it seems
to still gather quite a lot of traffic, so I think that it could be useful both for me and for potential readers to
get a new, up-to-date post about the tool!
The CFSSL version used in this post is v1.6.3
CFSSL is a toolkit built by Cloudflare, released in 2014.
It’s intended to be used to easily create, sign and serve TLS certificates from a small application which can be ran both locally and as a server (rest-ish json api).
The program is written in Go, which makes it easy to build yourself, or just download from GitHub for
most OS:es and architectures.
I personally prefer to run it as a docker container and use one of my own creations (shameless plug!).
The program have been used by cloudflare to generate their certificate chains, so from a “is it tested?” perspective, it feels quite sturdy.
Each TLS certificate consists of a public and a private certificate. The certificates can be “signed” by an authority, which makes
computers and other devices able to identify who issued the certificate and trust them (if they trust the root).
Generating a certificate without getting it signed makes it as much a certificate as if its signed, but each device which wants to
trust it will have to add it to their internal trust store.
So, the best way is to have a root certificate, which is made to sign other certificates, which can be added as a trusted root
hence all certificates signed by it will be as well.
When we build a certificate “chain”, it’s usually a good idea to create the root on a device which have no access to the internet, the device can then be destroyed (after creating an offline backup of the root and a bunch of child-certs) to keep the root as safe as possible.
There are hardware devices (HSM / Hardware security modules) which can be used to create a certificate more securely, but they are quite pricey and using a raspberry pi-zero or similar would probably be a lot cheaper if you intend to destroy it or stove away the raspberry after generating the certificate.
Each intermediate certificate will be able to create certificates as well, it’s sometimes even worth generating intermediates from the first intermediates, to create
a bigger chain and allowing you to easier rotate the certificates further down the chain if needed.
The certificate at the end of the chain is usually called a “leaf” certificate.
Certificate icons created by Smashicons - Flaticon
To even start using CFSSL we aught to install it. There are (as said earlier) multiple ways to install it, while, if you have go installed (whichever OS you use), you can get the latest version with a simple
# Newer go versions
go install github.com/cloudflare/cfssl/cmd/...@latest
# Older go versions
go get github.com/cloudflare/cfssl/cmd/...
That command will install all the tools included in cfssl, which might not be needed for your case.
If you are just testing the commands and want to see what happens, its totally okay to do it on your local computer, but if you intend to use the root certificate - that you are about to create - for more critical things, be sure to do it on a computer which is offline and won’t be connected to the network again. A production root certificate should be secure, and having it on a machine exposed to the net is not a good idea. If your root certificate runs off on the internet, you will have a HUGE headache and a lot of work to do to rotate all your certs!
To generate a new certificate with CFSSL we need to create a json file with the data that we want the certificate to have.
root.json
{
"CN": "Jitesoft CA",
"key": {
"algo": "ecdsa",
"size": 384
},
"CA": {
"expiry": "87660h",
"pathlen": 2
},
"names": [
{
"C": "SE",
"L": "Lund",
"O": "Jitesoft",
"ST": "Skania"
}
]
}
The above json includes the required data for a ECDSA root certificate for the Jitesoft CA.
The CN
property defines the certificates “common name”, the name of the “root”.
Depending on the usage, the CN should have different names, but in my case, I want my top-most certificate to be called
my company name and CA to make it known that it’s my certificate authority certificate.
The CA
clause allows us to define the pathlen
for the certificate as well as the expiry lifetime.
Default expiry for cfssl CA’s is 5 years, which might be enough, the example above uses 10 years though.
The pathlen variable indicates how many intermediate certificates that can be created in a hierachy below,
0 means that the CA can only sign the leaf certificates, 1 level of intermediates can be created, 2 that the intermediate
certificates can create sub intermediates and so on.
In a certificate used for a webserver, you would set the primary domain as the CN
, while you would add a
hosts
property (an array) with any alternative names (SAN) to make sure that the certificate is bound to the
specific domains only. But in the case of a CA, we rather want a generic name than a domain.
The key
property defines what type of key it is we want to generate, in this case, I have decided that my certificate
should be a ECDSA key with the size of 384 bits.
The final property, names
(subject names) gives anyone viewing the certificate a hint of the owner of the certificate.
C
= Country (ISO 3166-2 code), L
= Locality, O
= Organization and ST
= state.
If wanted, you may also include OU
(organizational unit name), as well as E
(email).
RSA and ECDSA are two algorithms which are quite commonly used for certificates, RSA is quite a lot older and well tested,
while ECDSA generates a lot smaller files and makes use of something called “Elliptic Curve Cryptography” (ECC).
ECDSA (or rather Elliptic Curve Digital Signature Algorithm) is a lot more complex than RSA (which instead of the curve makes use of prime numbers).
One issue with choosing an ECC algorithm is that there are software that does not “yet” (after 15+ years) support ECC algorithms. So you should choose an algorithm which is best suited for you to use and especially a size on the root which makes it secure enough (I would recommend using 2048 (or rather higher) with RSA and 384 and over with ECDSA).
The lowest RSA size which CFSSL will accept is 2048 and the highest is 8192, while it accepts 256, 384 as well as 512
while using an elliptic curve algorithm.
As of writing, the RSA and ECDSA algorithms are the only ones supported.
To create the certificate from the json data we created we invoke the
cfssl gencert
command.
cfssl gencert -initca root.json
Now, doing this will create a few values and output it to STDOUT in a json format, but by using the cfssljson
tool
we can parse it out into a set of files instead:
cfssl gencert -initca root.json | cfssljson -bare root
The gencert
command tells cfssl that we want to generate a new certificate (keys and sign request) and by using
the -initca option we also tell it that the certificate will be used for a certificate authority.
If you run the ls
command you should now find the following new files in the directory: root-key.pem
, root.csr
and root.pem
.
The root.pem
is your public key, this can be shared and uploaded everywhere as it’s not a secret (rather the other way around),
for a client to validate your signed messages, the certificate needs to be known, and this is done by “trusting” the public key.
The root.csr
will not be used with the root certificate, as in this example, we don’t use another CA to sign our certificate.
The root-key.pem
is a lot more critical that it does not slip out of your hands. This is the key which will be used to “prove”
that your CA is the CA actually signing the other certificates.
It will be used to generate the intermediate certificates and then hidden away.
With the help of openssl we can quickly verify our new certificate to make sure everything is correct:
> openssl x509 -in root.pem -noout -text
# Prints something like:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
61:c9:5c:9b:c2:28:32:41:3f:83:7d:ea:b8:82:65:0a:a3:ce:32:32
Signature Algorithm: ecdsa-with-SHA384
Issuer: C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft CA
Validity
Not Before: Jan 9 13:09:00 2023 GMT
Not After : Jan 9 23:09:00 2028 GMT
Subject: C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft CA
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (384 bit)
pub:
04:bc:18:70:4e:18:17:eb:4e:82:6e:b6:8f:83:e3:
c8:f3:85:27:a4:20:8f:d2:76:4e:38:9e:7b:6c:5f:
4f:ef:60:f8:f8:d1:52:a8:b8:b2:f7:a4:94:fa:f0:
cc:f9:c4:45:83:d5:52:29:4b:97:75:72:f3:a2:33:
ee:d8:e3:84:ae:bd:1b:a1:9a:54:71:9e:6e:1e:cc:
3c:83:ad:1d:78:c2:b5:9b:fb:69:52:ec:5c:79:24:
fd:48:9c:39:45:9c:22
ASN1 OID: secp384r1
NIST CURVE: P-384
X509v3 extensions:
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:2
X509v3 Subject Key Identifier:
72:28:B0:15:F5:62:F9:1D:17:CB:03:40:BB:B7:B8:AD:AA:A3:A4:A7
Signature Algorithm: ecdsa-with-SHA384
30:65:02:30:01:dd:5e:42:3e:fb:ef:cc:02:2c:ab:96:2d:06:
ee:95:fc:c7:22:ba:08:db:5d:b6:57:ba:95:0b:52:64:f7:37:
a5:c1:17:be:ee:ff:0a:87:35:0b:74:4d:1a:69:f6:21:02:31:
00:83:3d:01:67:d8:c1:f1:96:96:73:cf:00:6d:b3:60:b2:bf:
2d:05:e0:2e:ee:f7:09:40:41:c8:71:00:cc:b9:ff:31:d5:3e:
92:39:11:02:8d:1f:a2:37:a1:09:5f:8e:4e
As you can see, the public key shows the client the allowed functionality of the certificate (X509v3 extensions) as well as the information we supplied in the json file before.
When we have our root certificate we will want to create the intermediate certificates which we will later use to sign our leaf certificates with.
To keep our file structure a bit easier to handle as well as easier to display in a blog, create a subfolder for each new intermediate
to create.
In my case, I’ll create two: Jitesoft Intermediate 1
and Jitesoft Intermediate 2
and call the folders inter1
and inter2
to keep it simple.
mkdir inter1 inter2
CFSSL makes use of a profile concept for generation of new sub-certificates. The profiles configuration can be used for all kinds of certificates, while right now, we just create the intermediate profile:
profiles.json
{
"signing": {
"profiles": {
"intermediate": {
"usages": [
"cert sign",
"crl sign"
],
"ca_constraint": {
"is_ca": true
},
"expiry": "43800h"
}
}
}
}
Each profile requires a set of usages, you can as well define an expiry
here (which will replace the value set in the config.json file),
and for a Intermediate CA a ca_constraints
clause where we set is_ca
to true to indicate that it is actually a certificate authority
(which an intermediate certificate is).
For an intermediate authority, we need to se the usages cert sign
, crl sign
.
The cert sign usage allows the CA to sign certificates and the crl sign will allow the certificate to sign certificate revocations.
The profiles file is used when signing the certificates, and just as with the original ca file, we need a configuration file for the specific intermediates:
inter1/config.json
{
"CN": "Jitesoft Intermediate 1",
"key": {
"algo": "ecdsa",
"size": 384
},
"CA": {
"expiry": "43800h",
"pathlen": 1
},
"names": [
{
"C": "SE",
"L": "Lund",
"O": "Jitesoft",
"ST": "Skania"
}
]
}
With those two files, we can generate the intermediate certificate:
cd inter1
cfssl genkey -initca ./config.json | cfssljson -bare inter1
Inspecting the new intermediate certificate will show an unsigned certificate (which is basically the same as the CA) for a client to recognize that it’s issued by your CA, we need to sign it:
cfssl sign -ca ../root.pem -ca-key ../root-key.pem -profile intermediate --config ../profiles.json inter1.csr | cfssljson -bare inter1
In this case, we make use of the csr
file (certificate signing request), because we are actually requesting our certificate authority
to sign the certificate!
If you re-inspect the certificate with openssl, you will now see that the Issuer
have switched from the cert itself (Subject)
to the Subject of the CN:
> openssl x509 -in inter1.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
75:74:5e:57:04:c1:06:14:bb:bf:90:3c:93:20:36:bc:0f:38:3a:0d
Signature Algorithm: ecdsa-with-SHA384
Issuer: C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft CA
Validity
Not Before: Jan 9 13:27:00 2023 GMT
Not After : Jan 9 14:27:00 2023 GMT
Subject: C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft Intermediate 1
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (384 bit)
pub:
04:81:4d:6e:ea:b7:0b:c2:b0:80:06:3e:1b:22:9a:
84:6f:bc:aa:b5:24:bf:1d:83:4f:70:6f:12:bd:8e:
b0:27:cb:e5:7d:a7:8d:f6:da:d3:7d:9e:39:b0:95:
07:ae:fa:ad:58:33:72:d5:28:3b:e9:e0:b5:cb:1b:
82:2c:30:fa:ce:a7:ab:02:db:1b:a9:1e:15:c8:5a:
f8:cc:d2:c8:29:19:07:df:21:89:c6:60:56:b5:bc:
08:82:9a:b9:74:ab:5b
ASN1 OID: secp384r1
NIST CURVE: P-384
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Certificate Sign, CRL Sign
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
4A:25:E3:2D:62:83:BA:FF:37:3D:C4:A9:B7:13:00:3A:B4:8D:96:C5
X509v3 Authority Key Identifier:
keyid:72:28:B0:15:F5:62:F9:1D:17:CB:03:40:BB:B7:B8:AD:AA:A3:A4:A7
Signature Algorithm: ecdsa-with-SHA384
30:64:02:2f:0b:e0:46:e4:af:9f:86:23:35:dd:30:79:cd:af:
91:81:42:b7:cd:c7:90:d8:16:59:0d:43:b7:59:98:cc:65:6f:
45:17:74:b2:d9:ca:ef:c6:c8:1b:5e:51:62:fd:6d:02:31:00:
d5:d2:8c:50:be:37:00:15:31:d2:50:84:29:05:cc:d7:4b:17:
ef:49:8c:d1:6c:a3:5f:06:d1:b7:7d:b9:09:5b:f3:43:46:3e:
f4:11:16:80:c1:6a:10:8d:af:5a:91:e0
The same can be done with the inter2 to generate a second intermediate!
The whole reason to have a CA is of course to generate certificates, not just new CA:s, and those certificates
are the leaves.
Just as with the intermediate profile, the leaf certificate needs a profile with the usages
that it requires.
So, we can start with creating two types of certificates, one for server auth and one for client auth:
profiles.json
{
"signing": {
"profiles": {
"intermediate": {
"usages": [
"cert sign",
"crl sign"
],
"ca_constraint": {
"is_ca": true
},
"expiry": "43800h"
},
"server": {
"usages": [
"server auth"
],
"expiry": "720h"
},
"client": {
"usages": [
"client auth"
],
"expiry": "720h"
}
}
}
}
The two new clauses added are the profiles client
and server
.
In this example, I’ll create a new directory in the inter1 directory to keep the certificate hierarchy and folder structure as is:
mkdir inter1/certs
cd inter1/certs
We also need to create a configuration for the certificates:
inter1/certs/server.json
{
"CN": "Server",
"hosts": [
"127.0.0.1",
"server.domain",
"sub.domain.tld"
]
}
inter1/certs/client.json
{
"CN": "Client",
"hosts": [""]
}
As you see in the two configurations, we set a CN (common name), which - if this was a web certificate - would contain the primary domain of the page the certificate should be used for, and we add a hosts array, which contains a list of the IP-addresses that the certificate will actually be allowed for.
In this case, the certificates will be used for authentication, so the server have the addresses that it will be served on, while the client have an empty list, as we don’t want the certificate to be only used on one host.
To generate the certificates we - again - use the cfssl tool, but in this case without the -initca flag:
cfssl gencert -ca=../inter1.pem -ca-key=../inter1-key.pem \
-config=../../profiles.json \
-profile=server server.json | cfssljson -bare server
cfssl gencert -ca=../inter1.pem -ca-key=../inter1-key.pem \
-config=../../profiles.json \
-profile=client client.json | cfssljson -bare client
We can now inspect the certificates and see that they are signed with the correct certificate authority (Jitesoft Intermediate 1) that the usages are correct and that the SAN:s are correct:
> openssl x509 -in server.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
07:f9:b7:85:4f:12:a8:10:5c:16:dd:a0:b8:53:80:3c:3c:a4:97:e4
Signature Algorithm: ecdsa-with-SHA384
Issuer: C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft Intermediate 1
Validity
Not Before: Jan 9 14:10:00 2023 GMT
Not After : Feb 8 14:10:00 2023 GMT
Subject: CN = Server
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:52:b6:af:a7:db:dd:0d:2b:0f:ab:d6:49:c7:0e:
a8:eb:ef:29:ec:e4:b6:c1:cd:d3:0f:21:f4:5d:a3:
b0:ba:c9:b3:11:67:72:20:a7:ec:60:03:76:ec:b0:
08:30:14:6e:13:c5:52:66:2b:ec:d2:28:5d:cb:64:
a4:06:d9:af:e4
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
AA:83:51:38:F4:95:99:79:AB:3F:77:38:AF:77:CC:37:4A:C2:48:82
X509v3 Authority Key Identifier:
keyid:8C:D6:B7:3D:9E:0B:9B:5E:68:82:58:EC:91:84:27:89:FB:58:1B:6E
X509v3 Subject Alternative Name:
DNS:server.domain, DNS:sub.domain.tld, IP Address:127.0.0.1
Signature Algorithm: ecdsa-with-SHA384
30:65:02:30:7f:c4:45:f2:89:75:5d:ba:ec:32:1a:c8:bd:0a:
78:c5:c3:fa:86:d3:b9:cf:8d:6f:68:54:54:a1:23:5c:73:7d:
28:41:11:54:61:55:81:bb:03:5f:f0:be:c7:6a:d5:56:02:31:
00:bd:16:36:5e:2b:f5:1f:31:25:3c:00:bf:7d:86:fc:eb:91:
09:ae:05:23:31:8e:51:71:81:da:4b:14:1d:b2:95:16:25:8f:
9f:49:e8:b4:df:c5:08:dc:e9:d6:5d:cf:58
These certificates had a expiry
of 720 hours, so they will only be working for a month. This can ofcourse be changed
in the profiles.json file if you want longer certificates!
We can test the chain with openssl and cURL:
# In the `root` directory:
openssl s_server -cert ./inter1/certs/server.pem -key ./inter1/certs/server-key.pem -WWW -port 12345 -CAfile root.pem -verify_return_error -Verify 1
# Open a separate shell and enter the `root` directory:
curl -k --cert ./inter1/certs/client.pem --key ./inter1/certs/client-key.pem https://localhost:12345/test.txt
verify error:num=20:unable to get local issuer certificate
Oh now! This is not good!
This is because openssl (or any other server) can’t verify that the intermediate certificate is actually originating from the root CA. We actually need to bundle the certificates first.
This is done with one of the other tools which is supplied with cfssl, it’s called mkbundle
# in root directory:
mkbundle -f bundle.crt root.pem inter1/inter1.pem
We can cat our new bunlde to see the certificate bundle:
-----BEGIN CERTIFICATE-----
MIICMDCCAbagAwIBAgIUYclcm8IoMkE/g33quIJlCqPOMjIwCgYIKoZIzj0EAwMw
VjELMAkGA1UEBhMCU0UxDzANBgNVBAgTBlNrYW5pYTENMAsGA1UEBxMETHVuZDER
MA8GA1UEChMISml0ZXNvZnQxFDASBgNVBAMTC0ppdGVzb2Z0IENBMB4XDTIzMDEw
OTEzMDkwMFoXDTIzMDEwOTIzMDkwMFowVjELMAkGA1UEBhMCU0UxDzANBgNVBAgT
BlNrYW5pYTENMAsGA1UEBxMETHVuZDERMA8GA1UEChMISml0ZXNvZnQxFDASBgNV
BAMTC0ppdGVzb2Z0IENBMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEvBhwThgX606C
braPg+PI84UnpCCP0nZOOJ57bF9P72D4+NFSqLiy96SU+vDM+cRFg9VSKUuXdXLz
ojPu2OOErr0boZpUcZ5uHsw8g60deMK1m/tpUuxceST9SJw5RZwio0UwQzAOBgNV
HQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB/wIBAjAdBgNVHQ4EFgQUciiwFfVi
+R0XywNAu7e4raqjpKcwCgYIKoZIzj0EAwMDaAAwZQIwAd1eQj7778wCLKuWLQbu
lfzHIroI2122V7qVC1Jk9zelwRe+7v8KhzULdE0aafYhAjEAgz0BZ9jB8ZaWc88A
bbNgsr8tBeAu7vcJQEHIcQDMuf8x1T6SORECjR+iN6EJX45O
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIICWzCCAeCgAwIBAgIUTl02XWipAdn3Y8chGqzegLmrkhAwCgYIKoZIzj0EAwMw
VjELMAkGA1UEBhMCU0UxDzANBgNVBAgTBlNrYW5pYTENMAsGA1UEBxMETHVuZDER
MA8GA1UEChMISml0ZXNvZnQxFDASBgNVBAMTC0ppdGVzb2Z0IENBMB4XDTIzMDEw
OTEzNTYwMFoXDTI4MDEwODEzNTYwMFowYjELMAkGA1UEBhMCU0UxDzANBgNVBAgT
BlNrYW5pYTENMAsGA1UEBxMETHVuZDERMA8GA1UEChMISml0ZXNvZnQxIDAeBgNV
BAMTF0ppdGVzb2Z0IEludGVybWVkaWF0ZSAxMHYwEAYHKoZIzj0CAQYFK4EEACID
YgAEc9LuhhgVEa/Z1CXbYyshJPWjjHNGq8Q88rvU+inxfHCUr/5l10SvwIEaNHiD
FalwWmf/dEtfboPGfI2IaYZZ4A4S8CILK8q90JzpZkPZKpRrdwCSR8BLN3Q7YPVv
Hry0o2MwYTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4E
FgQUjNa3PZ4Lm15ogljskYQniftYG24wHwYDVR0jBBgwFoAUciiwFfVi+R0XywNA
u7e4raqjpKcwCgYIKoZIzj0EAwMDaQAwZgIxAPjNCQcRzPsAPudk0PM7I++B/ihk
kqBaVcVtl75Ru0qCr3T85QEZpQQd6xMLAhOe/QIxAPTwercxV4RwPusrvlLHAqI+
bu3IiUngL2bdz+vU1Pk2i8uzi9kWmL8KocVt+sKXWg==
-----END CERTIFICATE-----
This bundle is the CA
certificate that is added to any program which requires the CA.
Now, we just need to modify the openssl command to use the ca bundle instead of the root ca:
openssl s_server -cert ./inter1/certs/server.pem -key ./inter1/certs/server-key.pem -WWW -port 12345 -CAfile bundle.crt -verify_return_error -Verify 1
# Open a separate shell and enter the `root` directory:
curl -k --cert ./inter1/certs/client.pem --key ./inter1/certs/client-key.pem https://localhost:12345/test.txt
And we will have a successful response:
depth=2 C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft CA
verify return:1
depth=1 C = SE, ST = Skania, L = Lund, O = Jitesoft, CN = Jitesoft Intermediate 1
verify return:1
depth=0 CN = Client
verify return:1
FILE:file.txt
And that my friend, is how you set up your own certificate authority and chain.
As always, if you find any issues with the tutorial just let me know, and I’ll update it as soon as possible!
]]>I’ve actually created a bunch of drafts, but none have been published yet as they haven’t been finished…
So.. why? Whats up?
About 1½ years ago I was rushed to the ER. My doctor had called me to report
on some tests I had done for my WED, she was a bit
startled as the tests had shown that I had acute anemia.
This was kind of a shock, but at the least it explained why I was so tired and almost fained
from the smallest tasks (such as walking up the stairs). My blood value was at around 70, which is
around 90 units below my standard and I had to be filled up with two bags of blood before they sent me home.
It took a while for me to recover and I had to medicate for a while, but my blood value went up again and
the doctors could find no reason for why I was sick.
I went on scheduled tests (weekly, then bi-weekly and then every month) and after a while it returned from nowhere.
All tests re-started and nothing could be found.
As of now, I’m alright, my hB value is okay (over 150 again), my iron levels are still very low and I eat more iron than
most people have in their kitchen, but I’m okay.
The ordeal did though take quite a lot of my mental stamina (especially as we still don’t know why
I’ve been sick and I still have to go through a lot of testing and examinations).
My company is a single person company, so when I’m sick, there is no income, so most of the time when I’ve not been sick and in bed has been spent on working (and family of course). It goes well, a lot better than I had imagined due to all my problems, but it has forced me to put side-projects (such as this blog and a lot of my open-source stuff) to the side.
But! This is a new year. And a new year obviously means that stuff should be different (right?!), so in the spirit of that, I thought I’d try to get going with the blogging again, it might take some time, as I’m a very slow blogger, but I do hope to have a few new articles out soon.
I’ve been working on a new CFSSL blog post, something I have had requests on re-visiting and I still see a lot of hits on, further I thought I’d try to deep-dive in gRPC and micro-service architecture, something that interests me and I have been researching for a while.
And I guess that’s it, now you know!
]]>As a vivid user of PGP and other signature solutions, cosign is something I’ve been looking at for a while. I just recently started using it to sign container images to be able to implement a bit more strict procedure in my container building and usage flow, something that was a lot easier than I thought!
So, I think that part covers at the least the ‘What’ bit of the excerpt, so let’s head on to ‘Why’.
Most people working in tech have encountered signatures from time to time, it’s often a hazzle if you wish to verify stuff and to keep your own keys can be a real pain. The sigstore project have a few solutions to ease the process (fulcio), but this part will focus more on the signing and verifying bits.
When it comes to why someone would like to sign their images (or binaries or whatever they feel like signing), the reason is quite simple, it’s to allow for consumers to validate the resource and (as long as you are someone they trust) know that what they download, run or install is actually from you and is safe to use.
With my company, I try to make sure that we are both transparent and a source of trustworthy software, and when we release things, I want people to be able to
verify that it’s actually from us, not from someone using the name or even hijacking one of our accounts.
Signing and verification consists of two keys and a checksum, the private key is owned by the one who produces the resource, while the public key is to allow
validation. The checksum that is the actual signature only fits the private key and if it’s signed with the wrong key, the public key will not accept it as a valid
signature.
That way, the trust is in that the key is safe and not used by anyone but the developer while any binary uploaded anywhere can be validated to at the least that point.
There are occurrences when a key runs aloft and malicious software is published with a valid signature, but it’s a lot rarer than seeing an access token or similar get lost and used.
Keeping a “root” key secret is not always easy and depending on the risk of a lost key, there are a lot of different approaches one could take.
I would personally always recommend that one create their root key on a non-online machine and that the machine is wiped after generating the key and some intermediates, but when it comes to this tutorial, we will just use a fresh key and think about the paranoia later!
The first thing one have to do to be able to sign their images with cosign is to get ahold of the software.
Cosign can either be installed or ran through docker (or another container runtime). I will describe the ‘install’ way here rather than
using docker.
If you are a go user (which cosign, just as about every new non-frontend-web project now ‘a day is!) and have go 1.6+ installed, you can
install the latest version of cosign with a simple go install
command:
go install github.com/sigstore/cosign/cmd/cosign@latest
But if you prefer to use binaries, the easiest way is to download the binary from GitHub:
(Linux-ish way!)
# Depending on your architecture you might want to use something other than amd64
ARCH=amd64
LATEST=$(wget -qO- https://api.github.com/repos/sigstore/cosign/releases | jq -r ".[0].name")
wget https://github.com/sigstore/cosign/releases/download/${VERSION}/cosign-linux-${ARCH}
mv cosign-linux-${ARCH} /usr/bin/cosign
chmod +x /usr/bin/cosign
For Windows user, there are exe releases on the cosign release page on GitHub: https://github.com/sigstore/cosign/releases
Test to make sure that it’s installed correctly
cosign version
GitVersion: v1.2.1
GitCommit: unknown
GitTreeState: unknown
BuildDate: unknown
GoVersion: go1.16.6
Compiler: gc
Platform: linux/amd64
And initialize cosign (creates a .sigstore config directory in your home dir): cosign init
.
Now when we have cosign installed we can actually create our first key-pair.
This is simply done by invoking the cosign binary like this:
cosign generate-key-pair
Which produces a cosign.key
and a cosign.pub
file in the directory where you ran it.
The private key is supposed to be private, really private. If you lose your private key and someone gets a hold of it, they can
basically upload binaries with valid signatures in your name, something you really don’t want!
With that said, we can’t throw the key on a usb stick and forget about it for the next 100 years, no, we actually need the key to sign our images!
The public key is for your users. You can distribute it basically however you want, it’s public and totally okay to just throw into a gist or to upload on a warez site! No one can do much with it more than validating your payloads anyway!
So, when we got our two keys, we can basically start sign our images right away.
To sign an image, you use the cosign sign -key cosign.key my-org/my-image
, this will create a new signature and upload it to the registry.
To test if it’s a valid image, you use the cosign verify -key cosign.pub my-org/my-image
and you should get an output which looks something like
this:
Verification for index.docker.io/jitesoft/ubuntu:latest --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The signatures were verified against the specified public key
- Any certificates were verified against the Fulcio roots.
[{"critical":{"identity":{"docker-reference":"index.docker.io/jitesoft/ubuntu"},"image":{"docker-manifest-digest":"sha256:e2700dee042c018ed9505940f6ead1de72155c023c8130ad18cd971c6bfd4f03"},"type":"cosign container image signature"},"optional":{"sig":"jitesoft-bot"}}]
With a lill bit of JQ, you can get it pretty printed as well!:
Verification for index.docker.io/jitesoft/ubuntu:latest --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The signatures were verified against the specified public key
- Any certificates were verified against the Fulcio roots.
[
{
"critical": {
"identity": {
"docker-reference": "index.docker.io/jitesoft/ubuntu"
},
"image": {
"docker-manifest-digest": "sha256:e2700dee042c018ed9505940f6ead1de72155c023c8130ad18cd971c6bfd4f03"
},
"type": "cosign container image signature"
},
"optional": {
"sig": "jitesoft-bot"
}
}
]
As you can see in the payload above, there is an ‘optional’ object in the JSON in which I have added a ‘sig’ key. That’s called an annotation,
with annotations you can add any arbitrary data to the signature layer pushed to the registry.
Adding annotations is done by the -a
flag (can be used multiple times) and then a key=value as the flag value.
As many others, I don’t manually build my images, I use CI scripts on GitLab. To sign each and every new image published I’d have to spend most of my days signing images, something I do not like to do, so I want to have it as a part of my build script. The sigstore project supplies a cosign action for GitHub, which can be easily used by a single step:
uses: sigstore/cosign-installer@main
I don’t use GitHub for my pipelines though, so I decided to write my own template for GitLab.
The template can be easily extended from the Jitesoft gitlab-ci-template library if wanted,
or you could copy it and modify it after your own choice, as most of the stuff
I do outside of client work, it’s released under the MIT license.
The following script is the one that I use:
.sign:
image: registry.gitlab.com/jitesoft/dockerfiles/cosign:latest
variables:
COSIGN_ANNOTATIONS: ""
before_script:
- if [ -z ${COSIGN_PUB_KEY_PATH+x} ]; then echo "Failed to find public key"; exit 1; fi
- if [ -z ${COSIGN_PRIV_KEY_PATH+x} ]; then echo "Failed to find private key"; exit 1; fi
- |
if [ ! -z ${DOCKER_CRED_FILE+x} ]; then
mkdir ~/.docker
cp ${DOCKER_CRED_FILE} ~/.docker/config.json
fi
- |
if [ ! -z ${SIGN_IMAGES+x} ] && [ ! -z ${SIGN_TAGS} ]; then
wget https://gist.githubusercontent.com/Johannestegner/093e8053eabd795ed84b83e9610aed6b/raw/helper.sh
chmod +x helper.sh
COSIGN_TAGS=$(./helper.sh imagelist "${SIGN_IMAGES}" "${SIGN_TAGS}")
elif [ -z ${COSIGN_TAGS+x} ]; then
echo "Failed to find tags to sign"
exit 1
fi
script:
- |
for IMAGE in ${COSIGN_TAGS};
do
cosign sign ${COSIGN_ANNOTATIONS} -key ${COSIGN_PRIV_KEY_PATH} ${IMAGE}
done
OBSERVE if you run this script, only allow it on protected branches, and if possible, make sure you use your own runners. And as I have said many times in this post… DO NOT LOSE YOUR PRIVATE KEY!
As you can see, I do a bit more than just sign the image, so I’ll briefly explain how it works.
The image used is an image built (and signed of course!) under the jitesoft organisation.
It’s based on alpine linux to allow for a small-ish image but still the ability to run stuff interactively
(the official cosign image is a distroless image).
To skip using a docker run in the configuration, I use the image as the actual image the scripts run in.
It verifies that there are two environment variables (which I personally use gitlab secrets for) are set (should point to the path of the public and private key, even though the public key is not currently used) and then, if there is a docker credentials json file, it moves it to the home folder of the non-root user.
To make it even more easy for myself, I decided to include a small helper script in case the SIGN_IMAGES and SIGN_TAGS variables are set, it basically loops through all the images and gives them the same tags on all (as you can see in the helper.sh execution in the script).
When the tags are set up (or there already is a COSIGN_TAGS variable set) the script moves on to actually signing the tags with the cosign sign
command.
Any optional annotations is included with the COSIGN_ANNOTATIONS
variable.
I hope that this little tutorial gave some insight on why one would want to use cosign and similar tools and also how to use it in a simple way.
I will keep on using cosign and update or create a new post about it in the future, especially when I start testing fulcio in a larger scale!
]]>AArch64 is a part of Fosshost. Fosshost is a non-profit organisation that provides open-source projects with both x86_64 and aarch64 machines. My company, Jitesoft, uses both platforms to host a few of the GitLab runners that build our docker images.
AArch64 is a common name for the 64-bit version of the ARM architecture. All Jitesoft Docker images are built for both arm64 and x86_64 - and, if possible, others as well.
When we build an image with compiled binaries, we try to use the correct architecture for the builders. By doing that, we don’t have to set up a toolchain, and we don’t have to use Qemu or similar software to emulate the builds. When the binary is built, we mount it with the help of buildkit and copy it over to the image during the image build phase. This way, we can keep the image layers small without having to squash them and allowing us to build for many platforms.
Initially, we built them all during the image creation, with x86_64 machines only, but compiling larger projects using Qemu (which is, or at the least was at the time, single-core) made the compile times span over 10+ hours per architecture for some projects. Ten hours compile time was not feasible, so real Arm-based hardware was required.
When we started to build binaries for the ARM architecture, on real Arm-based hardware, the CI runners were deployed to machines at Scaleway (back then, they had pretty cheap Arm-based machines), but they ended this offering about a year ago. Linaro accepted us as a tenant on their ARM labs for a while, but they discontinued it a while back as well.
When we lost our runners at Linaro, we had to find something new, which was hard. Arm-based providers are not always cheap, and most of the ones we evaluated were quite a bit over our budget for an open-source project. We didn’t want to stop building the docker images, so a couple of Raspberry Pis were bought to run as dedicated CI servers.
RPi’s are pretty good for their price and size, but they are far from as powerful as a “real” machine, so the compilations were again taking way too much time. And then AArch64.com was launched!
The machine we run at AArch64.com is a lovely 8 core, 16GB RAM machine. The storage is sparse but, with a simple cronjob to clear the docker images stored on disk every now and then, that’s not an issue. But, we are using an IPv6 only network. I could have requested an IPv4 from Fosshost, but seeing that the machine doesn’t need an IP (more than for ssh), I figured it would not be necessary. Taking up an IPv4 for something that will just run without any access to the external network is not something one should do.
The machines themselves are already set up to use Cloudflares’ IPv6 DNS, so package downloading and such works out of the box! On our build servers, we only install the most necessary software: docker, the buildx docker-cli plugin, gitlab-runner (as GitLab is where we host our code, run our CI and initially publish our images) and any required dependencies. IPTables is set up to allow any outgoing traffic and only accept SSH in.
I’m not used to working with IPv6 at all! I wasn’t even aware of the fact that I didn’t have IPv6 at home. If a machine uses ipv6 and your ISP does not support it, you might have to connect through a so-called “jump-host” (which AArch64.com provides!), which I figured out quickly thanks to the documentation on the platform. Running Docker with IPv6, though… that took a bit more work to figure out!
Some people use Docker and IPv6, but reading through the net made me feel that it’s not something many people do. The steps to take are not extreme: the changes are made to the docker daemon, and some tweaks should be done to the firewall as well. It’s nothing complex, but without prior knowledge, it was a bit tricky!
First of all, the docker daemon has to be updated (create or edit the /etc/docker/daemon.json
file)
{
"ipv6": true,
"fixed-cidr-v6": "fd5f:a3e1:47c8:c8f4::/64"
}
The ipv6: true
flag will enable IPv6 in docker, but we also have to set a CIDR for the docker network to use. The value you add there should be a private ipv6 range; use any you wish, or even the above, as it was randomly generated! The size of 64 might not be needed, but a few IP addresses are available, so it should be quite fine.
Now, by restarting the docker daemon (systemctl restart docker
) the default network for docker should be using IPv6! This will enable IPv6 on the internal docker networks and for incoming/outgoing traffic.
Finally, we need to masq the traffic for docker. To do this, we need to add a postrouting rule on the IPv6 table, as docker does not do this itself:
ip6tables -t nat -A POSTROUTING -s fd00::/80 ! -o docker0 -j MASQUERADE
A simple reconfigure of the iptables-persist (or installation) and we are ready to reboot if wanted!
Our pipelines are set up to only share the docker socket on protected branches, branches only maintainers are allowed to merge and push to, so we allow our docker images to build privileged. This is unsafe if the builders are running on branches that are not protected. If you plan to do such things, make sure you read up on rootless docker and how that works, as it will allow you to configure docker without root privileges. Further, there are other OCI image builders do not require root, such as podman.
Protecting your servers and build environments against potential attacks is extremely important. If you are compromised and you accidentally publish images with malware or security holes, there are potentially thousands of people who might be at risk.
Building open-source is wonderful; the knowledge that people like and use the things you create is excellent. We are very grateful to Fosshost and the AArch64.com project for their contribution of server power to allow us to keep on doing it.
]]>I love programming, like… it’s what I have wanted to do for my whole life and even done
for most of it.
Back in 2016 I hit a wall, something that I never thought I would hit.
The wall hit me hard, it took me a year to recover enough to be able to start work again.
It started with panic attacks, at the time, I didn’t know it was panic attacks, as
I had never really had that before, I felt more and more depressed (didn’t really know that either).
It became more intense over time, and when I finally reached out to a doctor for help
I had multiple panic attacks each day, some so bad that I passed out at my desk at work.
I’ve had a few different jobs, some good (but less profitable) and some bad (still not that profitable), but the money was never my actual goal. I really don’t care that much about cash, as long as I have and get enough to survive and buy commodities now and then and especially as long as I have fun, the cash is less important. My best job was when I made about (equal to) $1600 per month, it was fun to work, I had the chance to do things good, and I really loved my co-workers and the company culture.
A few of the above things have happened at other companies. I have had co-workers I love
I have had fun (not always ofc) but… I have never felt that I’ve had the chance
to do what I can do, very rarely been allowed to do things good.
I’ve had bosses and team-leads who have pushed me down, made me feel that I can’t code,
I’ve had evaluations where I’ve been told that I’m not productive
(because of “superiors” who messed up every pull request I made),
I’ve been forced to build prototypes that I knew would end up in production as a bad product (never hits wrong).
All of that is quite standard in the business, while it’s something I can’t do without feeling bad.
So what did this do to me? Well, it made me feel like I was bad, like I didn’t
know what I was doing. Thinking back on it, I can see how it progressed.
I had to ask more about what I should do, how I should do it, (and even further down the line)
how things - I actually knew much about - worked.
I unlearned, I became worse in what I loved to do and what I actually knew how to do.
When I finally hit that wall, it took 3 months for me to be able to even use
a computer again.
It took me about 6 months before I could start write code again.
After 9 months or so, I finally understood that it wasn’t me who was a bad programmer
but rather the places where I worked that was the issue.
At that point I had quit my job, and after about a year, I went my own way with my
own company (and haven’t looked back).
From what I’ve seen, my progress was fast. A burnout can destroy someone for years or even for life.
I think I was lucky, and I think the fact that I just had my third
child, made a huge difference in how fast I recovered.
My burnout damaged me a lot. I still have issues, not only from the reasons why I got there and the medical problems it created (meds, therapy, all of that) but it also took a huge toll on my family and my self-esteem.
This post is not just something that I wrote to tell you a bit of my story, but especially a post to tell you that if you feel what I felt; just go, leave it, it’s not worth to hit a wall for someone - or something - who doesn’t even appreciate what you do.
]]>So, CIDR (Classless Inter-Domain Routing) is a concept for ip addresses and routing. It’s not new, first introduction was back in 1993, so most of us have probably heard of it or used the notation we are talking about here before, even though we might not really have thought too much about it.
When we define ip-ranges, we often use what is called “cidr notation”, and it looks something like this:
127.0.0.0/16
Where the first part (prefix) is the IP-Address and the last part (suffix) is the size of the range.
This notation is what I will try to explain in this post, it will get quite a bit technical, as it have a lot to do with bits and all that fun stuff, but bare with me and I’ll hopefully explain it in an easy way so that we all become experts on ip ranges (cause that’s totally what we all want!).
So, an IP address is basically a 32 bit integer. In this section I will focus on specific addresses and not ranges, as
an address makes use of the full integer (I will explain this a bit more later).
We will use the IP Address 192.168.0.1 in this description, a very common private IP address and probably used by one
or many of the routers that we use daily!
So, how can a 4-part dot-notation IP-address be a 32 bit integer value? Well, it’s quite simple…
All data types in their most basic form consists of bits. A bit is either on or off (0 or 1), when a data type is 32 bits
it contains 32 0’s or 1’s which can be changed to on or off to set the specific value. How it’s displayed to us humans
is just to make it easier to read.
So, a 32 bit integer can easily be split up in four parts, where each part is 8 bit (a byte), and that’s basically what an ip-address is:
[192]
byte one, [168]
byte two, [0]
byte three and [1]
byte four. Even if the value in the byte is smaller than what a full byte can use
it still have to take up a full byte for it to not be confusing for the machine reading it.
Each byte is - as said - 8 bits, so:
00000000 equals 0, all bits are off.
00000001 equals 1, all but one bit is off.
If we want the value 192, we get the bit value 11000000, and the full value 11111111 is naturally 255.
Putting the whole thing together, we got 4 byte units which for our IP-Address would look like this:
[11000000].[10101000].[00000000].[00000001]
or
192.168.0.1
An ip range still utilizes the bit that we use in the IP-Addresses, but we basically add on some values at the end of the 32 bit construct.
Depending on the size (in actual bits) of the range suffix, we make the prefix (the available ip-address) smaller or larger.
If we add 0 bit’s to the range, we have the whole net in an integer, it can range from 0.0.0.0 to 255.255.255.255, quite a large range.
If we add 1 bit (which is 1), we have halved the full range of the network 0.0.0.0 to 127.255.255.255.
Each new bit we add (say /4, 0100), will remove a chunk of the available addresses in the prefix, making it smaller and smaller, until we hit the bare minimum
of a /32 which only allows for a single IP-Address in the range.
Writing 192.168.0.1/32 is essentially the same as writing 192.168.0.1 but in CIDR notation.
So the sizes in a CIDR notated range is “easily” calculated as a backwards pow2:
CIDR | Ips | pow! |
---|---|---|
32 | 1 | 2 ⁽³² ⁻ ³²⁾ |
31 | 2 | 2 ⁽³² ⁻ ³¹⁾ |
30 | 4 | 2 ⁽³² ⁻ ²⁹⁾ |
29 | 8 | 2 ⁽³² ⁻ ³¹⁾ |
28 | 16 | 2 ⁽³² ⁻ ³⁰⁾ |
And | so | on… |
Now, you might think, but I can basically write 192.168.255.255/16 and that should just create some odd crazyness!
Well no. The /16 shifts away the first two bytes (8 + 8) making it basically say 192.168.0.0/16.
So when you overflow too far, say 192.168.0.0/4, you will lose the last 3 bytes, go lower than 2 and you will loose even more!
So, the CIDR notation keeps being a 32 bit value.
For ip 192.168.0.0
CIDR | Ips | Actual range |
---|---|---|
32 | 1 | 192.168.0.0 - 192.168.0.0 |
31 | 2 | 192.168.0.0 - 192.168.0.1 |
30 | 4 | 192.168.0.0 - 192.168.0.3 |
29 | 8 | 192.168.0.0 - 192.168.0.7 |
28 | 16 | 192.168.0.0 - 192.168.0.15 |
… | … | … |
16 | 65536 | 192.168.0.0 - 192.168.255.255 |
… | … | … |
8 | 16777216 | 192.0.0.0 - 192.255.255.255 |
… | … | … |
4 | 268435456 | 192.0.0.0 - 207.255.255.255 |
… | … | … |
1 | 2147483646 | 128.0.0.0 - 255.255.255.255 |
0 | 4294967296 | 0.0.0.0 - 255.255.255.255 |
So, why would we care? When will we ever use CIDR?
Well, I can’t really speak for everyone, but if you are interested in stuff like
kubernetes, or wish to set up a VPN or even a few VPS’es on your favorite provider, it might actually be quite good
to know the basics!
When you set up a kubernetes pod-network (with one of the network plugins that exists), it’s often possible (if not even needed!) to define your own pod and/or service network. It’s quite good to make sure that the network does not match up with the network that the machines communicate over!
Even your own home router might be a place where this could be needed. I personally have multiple sub-networks on my router, 192.168.0.0/20 is my main network, 192.168.16.0/20 could for example be a network space allocated for my guests, while I use 192.168.32.0/20 for my infrastructure through a VPN tunnel!
So, I wrote this blog post mainly during research of VPN for one of my customers, it’s quite fun and also quite advanced stuff on a first
glance, while after reading a bit, it makes a lot more sense.
If you read this and find any obvious errors or think that I misinterpreted anything, let me know! I’m not that awesome with bitstuff so
it’s for sure not impossible that I missed something!
That’s what I thought, pretty much, as I wanted to migrate a website to php8 (or at the least try to!).
The Easiest way to fix this is usually to just download it from pecl and go nuts, but at this time, there is no php8 binaries
on pecl for php8 imagick (from what I gathered, it doesn’t work with php8 yet?!).
So… how to fix..?
Well, first thing that one would check, is to see if someone else have compiled it already. I found a couple of links to file
stores which didn’t work or made me feel quite uncomfortable downloading files from…
So I thought that I’d compile it for myself.
Now, if my work computer was running linux, this would be a walk in the park, really, you just install a couple of packages and recompile php and it’s all done. But my computer is actually running Windows! So… now I had to figure out how to compile PHP extensions on a windows machine!
This is not the most easy task to be honest. I found a walkthrough on how to compile php and extensions on the php site.
It helped a bit, but seeing imagick is not a standard extension, it isn’t just a quick compilation away.
So, after having to struggle through a jungle of bad (or rather, none) documentation and after trying to “translate”
buildscripts for linux to windows, I finally got it working!
Due to the time I spent, I thought I’d write it down, both for my self (for future reference) and for anyone else who might
need to do the same thing in the future!
When you need to compile something you will need a few things before even being able to start with it. One thing I recommend
is to download visual studio (or at the least the build tools) so that you have all the c++ and c compilers and libraries that
are needed.
If you intend to build ImageMagick (that is, the actual program, not just the php extension) you should also
make sure that you install the windows sdk and latest MFC libs from the VS installer.
When all that is done, we need to create a base folder for building everything.
My C: disk is sacred, so I used my secondary disk, the D: disk!
To make everything simple, we will say that my base directory was D:\php-src
.
In this directory we aught to do a few things to make it possible to even start working on this, the first ting is to
install the PHP SDK tools.
The PHP SDK tools are used to build PHP source on windows (I’m sure it can be done in other ways, but this seems like an easy enough way).
You can find the source to the sdk tools at https://github.com/microsoft/php-sdk-binary-tools.
When you got the zip file from the releases at github or you have decided to clone the repo, you will want it extracted in the php-src
directory.
To make it easy, I used D:\php-src\sdk
for my directory.
You should now enter the directory in a terminal (preferably powershell or something, but I think anything will work) to allow the SDK tools to initialize the build directory structure.
cd D:\php-src\sdk
bin\phpsdk_buildtree.bat phpdev
This will create a new directory in D:\php-src\sdk
which will be named phpdev
. Inside the php-dev directory you will find a few folders.
Depending on your setup, you will have to locate the VS version (or VC) and the Architecture of the computer you are building on,
in case you use a standard setup and latest VS, you will have the path D:\php-src\sdk\phpdev\vs16\x64\
.
That’s where we aught to put the PHP source code.
When building the extension, you should make sure that the php version matches the one that you use yourself. I think that
build versions (or rather, patch versions in SEMVER) is okay to ignore, but I’m not fully sure about this. So to be safe, try to locate
the source package which corresponds to your version (maybe even update your own version while at it!).
You can find the php source at the same place where you download the actual binary packages: https://windows.php.net/download/
When you got the source at your given path (D:\php-src\sdk\phpdev\vs16\x64\php-8.0.2-src
or similar) you are ready to install the dependencies for
the base libs.
Open a new shell and enter the sdk directory.
From there, run the sdk bat file that corresponds with your vs version and arch (e.g., phpsdk-vs16-x64.bat) and you will enter a new shell.
While inside the shell, enter the php-8.0.2-src directory (the one containing the actual php code) and run the following command:
phpsdk_deps -u -b 8.0
This should create a few directories in the directory above where you currently are located. You may leave the php-sdk shell with the exit
command.
If you already have ImageMagick installed, you can skip this part.
If you wish to install ImageMagick from source, you can skip this part as well and head over to the next part.
But if you don’t, head over to the page and download the executable that you want (in my case it was the ImageMagick-7.0.11-1-Q16-HDRI-x64-dll.exe
) and install it at the wanted location.
When it is installed, you need to add a couple of environment variables to make sure that ImageMagick can be found.
Grab the path where you installed the binaries (in my case D:\Program files\ImageMagick
) and add it to your PATH
environment variable.
Also, while at it, add the variable MAGICK_HOME
with the same path as value.
Now, this will allow the linker to find the DLL’s that is needed to allow it to build the extension later on.
It’s not enough that we have the ImageMagick application installed, we actually also need the source and build it!
Head back over to the imagemagick page and download the sourcecode (https://imagemagick.org/script/install-source.php) and extract it in the php-src
directory (the one directly under D:
).
You now have to build the ImageMagick pre-configuration executable, which is a visual studio project located inside the D:\php-src\ImageMagick-<version>>\VisualMagick\configure
directory.
Open it in VS and allow it to upgrade the code to your version of the SDK if it asks for it.
With that done, build the solution and run it (can press the build and run button even) and after a short compilation time, it will open a prompt from which you can set up the
build directories. Make sure you tick the correct boxes in the Target Setup step, you will likely want the Static Multi-threaed DLL runtimes
(which, if you don’t have the ImageMagick program already installed, you will need), then you need to tick the Build 64-bit distribution if that is what you use.
Further on, you may configure the QD to fit the one you installed earlier (if you did), which in my case was the Q16
version.
Press next and it will allow you to set up where the files and dlls should be installed.
If you didn’t install the ImageMagick binary, anything that is generated in the future \bin\
dir, (which is the executables and the dlls) will have to be moved to your
desired installation path and then added to your path variable and MAGICK_HOME variable for it to be as we wish.
Don’t change the paths if you don’t really have to and just press next.
The configuration will create a new set of folders for you and a new Visual Studio solution, all located in teh VisualMagick
directory.
Open the solution and build it all (Build solution
).
When this is done (could take a few minutes), you will have successfully built ImageMagick from source (yay!).
Even if this could feel like a great success, we still have work to do, sorry…
Inside the VisualMagick
directory, we will now have a directory that we aught to use in the future configuration of the PHP source: lib
.
This is the library files that are required for the compiler to actually build the source, which is quite important.
Inside the ImageMagick
directory (which is in the same directory as the VisualMagick
directory) we will also find some stuff that we need, these are the
header files which we will need to include in the build as well.
The paths should be something like:
D:\php-src\ImageMagick-<version>\VisualMagick\lib
D:\php-src\ImageMagick-<version>\ImageMagick
With ImageMagick installed, we need to get a hold of the imagick extension source. Luckily, it’s available on github.
Get the source (clone or pick one of the zipped files under releases) and create a folder in the D:\php-src\sdk\phpdev\vs16\x64
directory. Name it pecl
and
extract the imagick source inside it (imagick should create a new directory, if it doesn’t just create a new imagick directory inside the pecl directory and move the files there).
Again run the php-sdk tool and re-enter the directory where the php source was extracted (D:\php-src\sdk\phpdev\vs16\x64\php-src
)
we now have to set up the configuration script for php, this is done with the: buildconf
command.
Buildconf generates a few files which allows us to run the configure
command, this command can be changed to build more (or rather in this case less) of
the php extensions and such that we need.
In this case, we don’t need anything but the imagick extension, so when we configure the build we disable everything else.
If you use a none-thread-safe version of php (yes it is important that it is compiled against the same target), you have to add the flag
--disable-zts
somewhere in the configure command. Other than that, the following would suffice:
configure --disable-all \
--enable-cli \
--with-imagick=shared \
# --disable-zts \ # Uncomment for none-thread safe!
--with-extra-includes="D:\php-src\ImageMagick-<version>\ImageMagick" \
--with-extra-libs="D:\php-src\ImageMagick-<version>\VisualMagick\lib"
This will disable all extensions that we don’t care about, it will also enable the imagick extension as a shared DLL, something we really want, and as well point the compiler to the given includes and libs that we need.
After the configuration is completed, follow the advice of the terminal and run the nmake
command.
Now, after nmake have finnished compiling our project, we can enter the <arch>/Release<-TS>
directory, inside this you will find the php_imagick.dll
file that you need!
Add it to your php ext
directory and enable it in the .ini file with a:
[Imagick]
extension=imagick
run php -m
and make sure that the imagick extension is a part of the output.
That’s it!
Compiling the extension was a lot more trouble than I actually thought it would be from start. Now..
it not that bad and it was mostly due to the fact that I had basically no information on what parts where needed
for the extension to compile, that is… a lot of trail and error!
I got it working, and hopefully I won’t have to bother with this again.
If you use windows for PHP, I would for sure recommend to just use a docker image or even PHP via WSL or similar. It’s easier
and works pretty much right away!
Hope the small writeup helps someone!
]]>I was asked a while ago, by the awesome guys at CIVO, if I wanted to do a talk on their community meetup.
Holding a talk/presentation is something I have never done before (except in school and such), but I said yes!
During the age of Corona, everything is done over the net, so it was not a meetup in person but over zoom. I’m quite used to distance work and collaboration over the net, so that was almost better than in-person (although it would have been fun to swim over to UK and say hi some day!).
But what would I even talk about? I have worked a lot with kubernetes, and I do know a lot about it both as a user and as a cluster manager… I have built multiple clusters from “the ground up”… Now… CIVO is managed k3s, so talking about setting up a cluster on your own servers feels not only useless, but I would feel like an arse too! hehe.
I think that one of the things I could actually be useful for is how I work with kubernetes, as a user!
And if I mix in a bit of CI/CD, something I work with on a daily basis, it would possibly be a decent talk!
So, I decided to go with Continuous deployment on kubernetes using GitOps on GitLab.
Now, I decided that the best way to do this is to actually go with something simple and basic, jumping in on high-level application management and showing of one of my 500+ lines of CI scripts would just confuse people who don’t work with this kind of stuff, so, basic is best.
Well, the meetup was awesome (see youtube clips at the bottom of this post!) and I feel that I did a decent job for a first talk ever (Although… I haven’t been able to listen to it myself and I give you the right to decide that for yourself, hehe), but I feel like I could expand a bit more on the whole thing in a blog post!
So here it is, let’s start from the beginning with…
Setting up a kubernetes cluster is not an easy task, so I would really recommend using a managed service, at the least for more vital sites and services!
Civo and their #kube100 beta is a good starting point if you are new to kubernetes, it’s an easy setup and there is a lot of things
you get “for free” with their marketplace and setup process.
This post will use a Civo cluster as the example cluster, and if you wish to read more about how to get started with CIVO, make sure you go read
my post on it here!
With CIVO’s new CLI, we can easily create a new kubernetes cluster with a short command:
civo kubernetes create my-awesome-cluster --size g2.medium --nodes 5 --save --wait --switch
The flags does the following: --size
sets what plan we want to use on our nodes, g2.medium is default, so the flag is not really needed, --nodes
set the node count,
3 is default, but we want a sturdy cluster with lots’a nodes! --save
, --wait
and --switch
makes sure that the CLI waits til the cluster is up and running (usually takes about a minute)
and then saves the configuration to your global kube file, it also switches to the new context allowing you to start work with the cluster right away!
There are more flags to use, check them out when playing around with the CLI, some might be really useful for your case!
When we deploy to kubernetes we will need a user that is able to update deployments. We won’t really change anything other than the containers in the deployment, but we will still need a “user” for it in the kubernetes cluster.
The most secure way to do this is to create a new Role and ServiceAccount which is bound to the namespace where the deployment is available. I usually create a deployer
account on each namespace that requires automatic or bot-controlled deployments:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: deploy-role
namespace: default
rules:
- apiGroups: ["apps"]
resources:
- deployments
- replicasets
- pods
- services
verbs:
- get
- update
- patch
- list
- watch
For security, I would personally recommend to remove the services resource and also the get, list and watch verbs, but it all depends on how much power you want to give to the role!
One important thing to notice is the apiGroups we allow the role to tamper with. That is, the apps
api group.
A kubernetes cluster have a whole lot of api groups, but under apps
we find the one we are looking for in this instance: apps.deployments
, we allow the role
to get, update, patch watch and also list all apps
resources that are defined under the resources
clause, that means, it can’t create new ones
and it can’t delete any.
apiVersion: v1
kind: ServiceAccount
metadata:
name: deployer
namespace: default
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: deploy-agent-binding
roleRef:
kind: Role
name: deploy-role
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: deployer
namespace: default
The above resources creates a new ServiceAccount
for the deployer. As you might see, it’s namespaced as well, that binds the account
to the default
namespace. A RoleBinding
resource is used to tie the Role
to the ServiceAccount
.
Now, when the SA and Role resources are deployed to the cluster, the deployer user will be able to log on to the cluster and update the deployments in the default namespace.
With the new service account up and running, we need to fetch the token and certificate that the account uses.
> kubectl apply -f deployer.yaml
# Fetch the account token name of the deployer user (will be something like 'deployer-token-lp6sf).
export KUBE_SERVICE_ACCOUNT=$(kubectl get serviceaccounts deployer -o json | jq -r ".secrets[0].name")
# Obtain the certificate and the token that will be used.
export KUBE_CA_CRT=$(kubectl get secret $KUBE_SERVICE_ACCOUNT -o json | jq -r '.data."ca.crt"')
export KUBE_TOKEN=$(kubectl get secret $KUBE_SERVICE_ACCOUNT -o json | jq -r ".data.token" | base64 -d)
With the token and the certificate we can easily create a kubeconfig
file for our new deploy user:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: $KUBE_CA_CRT
server: https://$KUBE_API_URI:6443
name: civo-k3s-demo
contexts:
- context:
cluster: civo-k3s-demo
user: civo-k3s-demo
namespace: default
name: civo-k3s-demo
current-context: civo-k3s-demo
kind: Config
preferences: {}
users:
- name: civo-k3s-demo
user:
token: $KUBE_TOKEN
user: $KUBE_SERVICE_ACCOUNT
This configuration we will later upload to the CI/CD runners, which is why we really want to make sure it only have access to just what it should and not your whole cluster.
There are loads of different “engines” for CI/CD, I prefer to use the ones that are directly connected to the git provider I use, but there are others that are really nice too. You will have to evaluate them for yourself to decide on which you feel most fit your needs. In my case, GitLab was the best choice.
GitLab have it’s own automatic DevOps setups which are really nice, especially if you connect your cluster directly to GitLab. This is not how I roll, I always need to have full control over everything… hehee
I will use GitLab type YAML configurations in this post, as that is what I write the best, but I will try to keep them as simple and easy to explain as possible.
So, when you have an engine for your jobs, you should be able to add a file or variable as a “secret”. That is
the var/file should not be possible - for people who does not have admin access to the jobs - to extract.
This is where we upload the kubeconfig.
If it’s possible to upload it as a file, make sure you create an environment variable pointing to the file path
and name it KUBECONFIG
so that we can easily access it.
If it’s a variable we go with, one easy way to do it is to add it as a b64 string and decode it into a file in a command in the ci job.
Either way, at the end, we should have a configuration file ready for when we need to deploy from the pipeline.
I wanted to take a short moment to describe how I think when I work with CI/CD pipelines.
Pipelines are aromatization jobs which should do the stuff that you don’t want to do by yourself every time.
When I build my pipelines, I always start out with writing them as I would do it manually, every single step I would
use if I was doing it at my local computer should be added to the pipeline.
The easiest way to do this - in my opinion - is to either write it all down, step by step, in a dockerfile.
Each step should have its own RUN
so that you can cache all the working steps, and then I test, and add and test
until it works as if I was doing it by hand.
Now, at this point the script is usually not optimized at all. It will likely take more time to run than I would want it to
in a production setting, so after I have a working script, I start refining it.
When refining, I don’t always write it as I would do it manually, while, I try to keep it as close to my personal
manual workflow in the terminal as possible.
Refining is a never ending task so when it’s good enough, it’s usually time to stop!
There are many ways to write pipelines, I will focus on the way I do it, but if wanted, a pipeline could
basically be a single script which runs when you push to your git repository.
Most CI/CD Engines have some type of stage system, that is, a way to define which jobs runs
in what order. Most also have some way of storing artifacts between the jobs. You could run a single job with all your
workload in it, but I personally find it a lot better to split it up, and allow jobs which can run parallel do that while
the jobs which have dependencies will wait for its turn.
So in my scripts I usually make use of the following basic stages:
You could use more, or less, it’s all up to you!
What does the different stages include? Well, if the names doesn’t explain themselves…
Test: This is the stage where all the code is tested. This could be standard unit tests or stuff like dependency scans
or code analysis.
Build: In this stage, the code is built for production, an artifact is created from each job and is then later used in the
containerize
jobs.
Containerize: Deploying to kubernetes without a image too assign to the containers is hard… So this step is used
to actually build the images and to push them to the registry that is used.
Scan: In the test
stage, scanning of dependencies and such is useful, but when you have your container ready, its
usually a good idea to scan the image too. This way you might find issues which are due to the base image you use or some
binaries that the image include. I personally use Trivy for my scans, but there are a lot of others on the market.
Deploy: As a final stage, the new build should be deployed.
All of the above stages are not really required to run on every branch or every pull request. That would be quite odd. Deployment for example, we don’t want to run on any other branches than the ones that we actually use for deployments, containerization and image scan is not really needed on all environments either, while the test job probably should run on all merge requests and all pushes to a branch.
My standard table for this is can be seen in the following table:
stage | branches |
---|---|
test | all + pr |
build | develop, feature-branch, master |
containerize | develop, feature-branch, master |
scan | develop, master |
deploy | develop, master + tag |
My environment uses three different type of deployments: dev
, stage
and prod
.
Production deploys are only done on tags on the master branch. Develop deploys on the dev branch (when pushed) and
stage deploys are made when something is pushed to the master branch.
Feature branches are not deployed, but images are built for them so that they can be tested locally or deployed to the cluster in case it is needed.
We won’t talk much about the dev and stage environments, rather just focus on the master branch, but having a decent strategy for testing before deploying to production could be a good thing to consider!
So, to slim the whole process down, the deploy script that I will explain in this post will use the following stages:
All the jobs will presume that there is only a master branch in the repository. And production deploys will require a tag to be created.
Testing is usually quite an important part of the deployment flow. Some tests are less important than others, but I would say that we always will want to stop the build if such as unit tests doesnt pass.
As an example I will break down a simple unit test job in the format that I use on gitlab for nodejs applications:
npm.test:
stage: test
image: jitesoft/node:latest
script:
- npm ci
- npm run test
All this job does is to tell the CI engine that the job is being ran in the test
stage, it uses a node image (one of mine!) and runs the following two
commands: npm ci
and npm run test
.
This is basically the same thing we would do in the terminal when testing our code, right?
Building an application could mean quite a bit of different things. Some applications have really complex build scripts, while
some are a simple one line.
We will stick to the node application example to make use of a more simple build script.
Presume that the package.json have a build:prod
script defined, which run all the required steps used to build the application.
npm.build.app:
stage: build
image: jitesoft/node:latest
variables:
NODE_ENV: development
script:
- npm ci
- npm run build
artifacts:
paths:
- dist/
npm.build.modules:
stage: build
image: jitesoft/node:latest
variables:
NODE_ENV: production
script:
- npm ci
artifacts:
paths:
- node_modules/
In the above case, the job has actually been split up into two jobs. This is so that we can run the whole process a bit faster (as they will run at the same time).
The first job will build our application code and place it in the dist
directory, the variables
property defines
the NODE_ENV
environment variable to use the development
environment. This because we most likely have a bunch
of packages which are required to build the code in the devDependencies
of the package.json file.
It builds the code into the dist folder and we mark the dist folder as an artifact (which will be uploaded to the CI server).
The second job will not build the app, but rather just install the production dependencies so that they can be used when running the application in
production. Just as with the dist
directory, an artifact is created for the node_modules
directory. The environment is set to production
as we don’t want to package all development dependencies and add them to the deployed application.
Containerizing an application is quite easy if you have ever used docker. All we need to do is to create an image with
the files that we want to deploy.
We will stick to the earlier node app to keep some type of red line in this, so the basic script would look something like this:
containerize:
stage: containerize
needs:
- npm.build.app
- npm.build.modules
stage: containerize
image: registry.gitlab.com/jitesoft/dockerfiles/misc:latest
script:
- docker buildx build -t ${CI_REGISTRY_IMAGE}:latest --build-arg VERSION=${CI_COMMIT_TAG} -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG} --push --platform linux/amd64 --progress plain .
rules:
- if: $CI_COMMIT_TAG
This containerization script uses an image which includes docker and the buildx cli plugin. I often use the buildx plugin so that I can build for more than one platform/architecture and it also gives us a few extra niceties to use when it comes to experimental syntax.
If you wish to read up more about the buildx plugin, feel free to check out this post!
When the image is built, we tag it with the latest
and tag
of the current pipeline. The rules
property only tells the gitlab ci scheduler that the job should
only run on tags (where the $CI_COMMIT_TAG
variable will be auto-populated by the engine to contain the current tag name).
The job have a needs
clause defined so that all the artifacts from the jobs in the list are downloaded to the job!
Ofcourse, all of this requires a dockerfile, and the most simple one we could go with is a node image of some sort:
FROM jitesoft/node-base:latest
RUN mkdir /usr/local/app
ADD ./dist /usr/local/app/dist
ADD ./node_modules /usr/local/app/node_modules
ADD ./package.json /usr/local/app/
EXPOSE 3000
CMD ["/usr/local/bin/node", "/usr/local/app/"]
The image only copies the dist, node_modules and package.json files to the image and defines the CMD and exposes the port that is used
by the application.
The image should be tested locally a couple of times, so that we haven’t messed something up, but if it runs as it should, it’s ready to be built!
What would a kubernetes deployment be without the actual deployment file?
That’s a good question, probably not much fun - would be my answer. So, at this point in the deployment process
or rather in the process of writing a deploy script, we should run the initial deployment, that is, we want the pipeline
to create an image that we can use when we deploy the application.
You could start off the deployment by just deploying another container instead, but I prefer to do a manual deploy with the real
application as a first deploy, so that I know everything goes right and it works as intended.
So, the deployment would look something like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-awesome-app
namespace: default
spec:
selector:
matchLabels:
app: my-app
env: production
project: my-project
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
spec:
containers:
- name: my-node-container
imagePullPolicy: Always
image: registry.whereever.com/my-node-app:latest
resources:
requests:
cpu: 30m
memory: 64Mi
limits:
cpu: 300m
memory: 512Mi
ports:
- containerPort: 3000
name: web
This deployment uses the image that we just created in the registry, it have limits and requests set for resources (something you should always do!) and
it is placed in the correct namespace (this is important, as our deployer only have access to the default namespace).
As you might notice, we are using 3 replicas, and we only allow 1 to be unavailable at a time. This will create
three identical pods with the same image and resources as defined in the template spec, so take all the requests
and limits
resources and
multiply with three to see how much the cluster will require of a node to be able to schedule it successfully.
When you feel ready, you can deploy the deployment with kubectl:
kubectl apply -f my-deployment.yml
You will ofcourse require a few more resources than just the deployment to be able to access it, such as a Service and some type of ingress. But I’ll not get in to that here!
Now we have a deployment up and running, all that is left is to create the actual deploy script in the CI pipeline.
A really simple deploy script could look something like this:
.deploy:
stage: deploy
image: jitesoft/kubectl:latest
variables:
IMAGE: "registry.whereever.com/my-node-app:${$CI_COMMIT_TAG}"
script:
- kubectl set image deployment.apps/my-awesome-app my-node-container=${IMAGE}
rules:
- if: $CI_COMMIT_TAG
Just as with the containerization script in the earlier stage, we will only run this on tags.
The image used here is a standard kubectl image and the command we invoke is basically
just a patch request of the deployment so that it changes the image to the latest build image.
In this stage, we really need to be able to access that kubeconfig file we created for the deployer earlier. So,
you can either add it as a secret file, if the CI/CD engine allows for such (in that case, you will want an env
variable to point to the file-path which is named KUBECONFIG
), or you could add the whole file as a variable and
then echo
it into a file, which you then export a env variable for (KUBECONFIG=/path/to/config
), just be sure to
remove the file afterwards, so that no one else can use it!
When all is ready, just push the new CI file to the repository and add a tag to see how the deployment job runs!
Thats it! It’s a fully working awesome auto-deployment script! Every time you push a tag to the master branch, your new pipeline will kick in and test, build, containerize and finally deploy the image to your kubernetes cluster!
CI/CD and Kubernetes is a huge field, there are sooo much that could be done in so many ways, the one I describe in this post might not fit all, honestly, for some projects it doesn’t even fit me, but it is a simple enough (while still functional) deployment pipeline.
I hope you enjoyed the post!
Youtube - Me on Civo community meetup
Youtube - The whole Civo community meetup