Kube deploys! My CI/CD process.

Introduction

I was asked a while ago, by the awesome guys at CIVO, if I wanted to do a talk on their community meetup.
Holding a talk/presentation is something I have never done before (except in school and such), but I said yes!

During the age of Corona, everything is done over the net, so it was not a meetup in person but over zoom. I’m quite used to distance work and collaboration over the net, so that was almost better than in-person (although it would have been fun to swim over to UK and say hi some day!).

But what would I even talk about? I have worked a lot with kubernetes, and I do know a lot about it both as a user and as a cluster manager… I have built multiple clusters from “the ground up”… Now… CIVO is managed k3s, so talking about setting up a cluster on your own servers feels not only useless, but I would feel like an arse too! hehe.

I think that one of the things I could actually be useful for is how I work with kubernetes, as a user!
And if I mix in a bit of CI/CD, something I work with on a daily basis, it would possibly be a decent talk!

So, I decided to go with Continuous deployment on kubernetes using GitOps on GitLab.

Now, I decided that the best way to do this is to actually go with something simple and basic, jumping in on high-level application management and showing of one of my 500+ lines of CI scripts would just confuse people who don’t work with this kind of stuff, so, basic is best.

Well, the meetup was awesome (see youtube clips at the bottom of this post!) and I feel that I did a decent job for a first talk ever (Although… I haven’t been able to listen to it myself and I give you the right to decide that for yourself, hehe), but I feel like I could expand a bit more on the whole thing in a blog post!

So here it is, let’s start from the beginning with…

Getting the cluster running!

Setting up a kubernetes cluster is not an easy task, so I would really recommend using a managed service, at the least for more vital sites and services!
Civo and their #kube100 beta is a good starting point if you are new to kubernetes, it’s an easy setup and there is a lot of things you get “for free” with their marketplace and setup process.
This post will use a Civo cluster as the example cluster, and if you wish to read more about how to get started with CIVO, make sure you go read my post on it here!

With CIVO’s new CLI, we can easily create a new kubernetes cluster with a short command:

civo kubernetes create my-awesome-cluster --size g2.medium --nodes 5 --save --wait --switch

The flags does the following: --size sets what plan we want to use on our nodes, g2.medium is default, so the flag is not really needed, --nodes set the node count, 3 is default, but we want a sturdy cluster with lots’a nodes! --save, --wait and --switch makes sure that the CLI waits til the cluster is up and running (usually takes about a minute) and then saves the configuration to your global kube file, it also switches to the new context allowing you to start work with the cluster right away!

There are more flags to use, check them out when playing around with the CLI, some might be really useful for your case!

Setting up all the important stuff

When we deploy to kubernetes we will need a user that is able to update deployments. We won’t really change anything other than the containers in the deployment, but we will still need a “user” for it in the kubernetes cluster.

The most secure way to do this is to create a new Role and ServiceAccount which is bound to the namespace where the deployment is available. I usually create a deployer account on each namespace that requires automatic or bot-controlled deployments:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deploy-role
  namespace: default
rules:
  - apiGroups: ["apps"]
    resources:
      - deployments
      - replicasets
      - pods
      - services
    verbs:
      - get
      - update
      - patch
      - list
      - watch

For security, I would personally recommend to remove the services resource and also the get, list and watch verbs, but it all depends on how much power you want to give to the role!

One important thing to notice is the apiGroups we allow the role to tamper with. That is, the apps api group.
A kubernetes cluster have a whole lot of api groups, but under apps we find the one we are looking for in this instance: apps.deployments, we allow the role to get, update, patch watch and also list all apps resources that are defined under the resources clause, that means, it can’t create new ones and it can’t delete any.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: deployer
  namespace: default
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: deploy-agent-binding
roleRef:
  kind: Role
  name: deploy-role
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: deployer
    namespace: default

The above resources creates a new ServiceAccount for the deployer. As you might see, it’s namespaced as well, that binds the account to the default namespace. A RoleBinding resource is used to tie the Role to the ServiceAccount.

Now, when the SA and Role resources are deployed to the cluster, the deployer user will be able to log on to the cluster and update the deployments in the default namespace.

With the new service account up and running, we need to fetch the token and certificate that the account uses.

> kubectl apply -f deployer.yaml

# Fetch the account token name of the deployer user (will be something like 'deployer-token-lp6sf). 
export KUBE_SERVICE_ACCOUNT=$(kubectl get serviceaccounts deployer -o json | jq -r ".secrets[0].name")

# Obtain the certificate and the token that will be used.
export KUBE_CA_CRT=$(kubectl get secret $KUBE_SERVICE_ACCOUNT -o json | jq -r '.data."ca.crt"')
export KUBE_TOKEN=$(kubectl get secret $KUBE_SERVICE_ACCOUNT -o json | jq -r ".data.token" | base64 -d)

With the token and the certificate we can easily create a kubeconfig file for our new deploy user:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: $KUBE_CA_CRT
    server: https://$KUBE_API_URI:6443
  name: civo-k3s-demo
contexts:
- context:
    cluster: civo-k3s-demo
    user: civo-k3s-demo
    namespace: default
  name: civo-k3s-demo
current-context: civo-k3s-demo
kind: Config
preferences: {}
users:
  - name: civo-k3s-demo
    user:
      token: $KUBE_TOKEN
      user: $KUBE_SERVICE_ACCOUNT

This configuration we will later upload to the CI/CD runners, which is why we really want to make sure it only have access to just what it should and not your whole cluster.

Different CI/CD engines

There are loads of different “engines” for CI/CD, I prefer to use the ones that are directly connected to the git provider I use, but there are others that are really nice too. You will have to evaluate them for yourself to decide on which you feel most fit your needs. In my case, GitLab was the best choice.

GitLab have it’s own automatic DevOps setups which are really nice, especially if you connect your cluster directly to GitLab. This is not how I roll, I always need to have full control over everything… hehee

I will use GitLab type YAML configurations in this post, as that is what I write the best, but I will try to keep them as simple and easy to explain as possible.

So, when you have an engine for your jobs, you should be able to add a file or variable as a “secret”. That is the var/file should not be possible - for people who does not have admin access to the jobs - to extract.
This is where we upload the kubeconfig.

If it’s possible to upload it as a file, make sure you create an environment variable pointing to the file path and name it KUBECONFIG so that we can easily access it.
If it’s a variable we go with, one easy way to do it is to add it as a b64 string and decode it into a file in a command in the ci job.

Either way, at the end, we should have a configuration file ready for when we need to deploy from the pipeline.

CI

What are you thinking?!

I wanted to take a short moment to describe how I think when I work with CI/CD pipelines.
Pipelines are aromatization jobs which should do the stuff that you don’t want to do by yourself every time.
When I build my pipelines, I always start out with writing them as I would do it manually, every single step I would use if I was doing it at my local computer should be added to the pipeline.
The easiest way to do this - in my opinion - is to either write it all down, step by step, in a dockerfile. Each step should have its own RUN so that you can cache all the working steps, and then I test, and add and test until it works as if I was doing it by hand.

Now, at this point the script is usually not optimized at all. It will likely take more time to run than I would want it to in a production setting, so after I have a working script, I start refining it.
When refining, I don’t always write it as I would do it manually, while, I try to keep it as close to my personal manual workflow in the terminal as possible.
Refining is a never ending task so when it’s good enough, it’s usually time to stop!

The stages!

There are many ways to write pipelines, I will focus on the way I do it, but if wanted, a pipeline could basically be a single script which runs when you push to your git repository.
Most CI/CD Engines have some type of stage system, that is, a way to define which jobs runs in what order. Most also have some way of storing artifacts between the jobs. You could run a single job with all your workload in it, but I personally find it a lot better to split it up, and allow jobs which can run parallel do that while the jobs which have dependencies will wait for its turn.

So in my scripts I usually make use of the following basic stages:

• test
• build
• containerize
• scan
• deploy

You could use more, or less, it’s all up to you!

What does the different stages include? Well, if the names doesn’t explain themselves…

Test: This is the stage where all the code is tested. This could be standard unit tests or stuff like dependency scans or code analysis.
Build: In this stage, the code is built for production, an artifact is created from each job and is then later used in the containerize jobs.
Containerize: Deploying to kubernetes without a image too assign to the containers is hard… So this step is used to actually build the images and to push them to the registry that is used.
Scan: In the test stage, scanning of dependencies and such is useful, but when you have your container ready, its usually a good idea to scan the image too. This way you might find issues which are due to the base image you use or some binaries that the image include. I personally use Trivy for my scans, but there are a lot of others on the market.
Deploy: As a final stage, the new build should be deployed.

All of the above stages are not really required to run on every branch or every pull request. That would be quite odd. Deployment for example, we don’t want to run on any other branches than the ones that we actually use for deployments, containerization and image scan is not really needed on all environments either, while the test job probably should run on all merge requests and all pushes to a branch.

My standard table for this is can be seen in the following table:

My environment uses three different type of deployments: dev, stage and prod.
Production deploys are only done on tags on the master branch. Develop deploys on the dev branch (when pushed) and stage deploys are made when something is pushed to the master branch.

Feature branches are not deployed, but images are built for them so that they can be tested locally or deployed to the cluster in case it is needed.

We won’t talk much about the dev and stage environments, rather just focus on the master branch, but having a decent strategy for testing before deploying to production could be a good thing to consider!

So, to slim the whole process down, the deploy script that I will explain in this post will use the following stages:

• test
• build
• containerize
• deploy

All the jobs will presume that there is only a master branch in the repository. And production deploys will require a tag to be created.

Test

Testing is usually quite an important part of the deployment flow. Some tests are less important than others, but I would say that we always will want to stop the build if such as unit tests doesnt pass.

As an example I will break down a simple unit test job in the format that I use on gitlab for nodejs applications:

npm.test:
  stage: test
  image: jitesoft/node:latest
  script:
    - npm ci
    - npm run test

All this job does is to tell the CI engine that the job is being ran in the test stage, it uses a node image (one of mine!) and runs the following two commands: npm ci and npm run test.
This is basically the same thing we would do in the terminal when testing our code, right?

Build

Building an application could mean quite a bit of different things. Some applications have really complex build scripts, while some are a simple one line.
We will stick to the node application example to make use of a more simple build script.

Presume that the package.json have a build:prod script defined, which run all the required steps used to build the application.

npm.build.app:
  stage: build
  image: jitesoft/node:latest
  variables:
    NODE_ENV: development
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/

npm.build.modules:
  stage: build
  image: jitesoft/node:latest
  variables:
    NODE_ENV: production
  script:
    - npm ci
  artifacts:
      paths:
        - node_modules/

In the above case, the job has actually been split up into two jobs. This is so that we can run the whole process a bit faster (as they will run at the same time).

The first job will build our application code and place it in the dist directory, the variables property defines the NODE_ENV environment variable to use the development environment. This because we most likely have a bunch of packages which are required to build the code in the devDependencies of the package.json file.
It builds the code into the dist folder and we mark the dist folder as an artifact (which will be uploaded to the CI server).

The second job will not build the app, but rather just install the production dependencies so that they can be used when running the application in production. Just as with the dist directory, an artifact is created for the node_modules directory. The environment is set to production as we don’t want to package all development dependencies and add them to the deployed application.

Containerize

Containerizing an application is quite easy if you have ever used docker. All we need to do is to create an image with the files that we want to deploy.
We will stick to the earlier node app to keep some type of red line in this, so the basic script would look something like this:

containerize:
  stage: containerize
  needs:
    - npm.build.app
    - npm.build.modules
  stage: containerize
  image: registry.gitlab.com/jitesoft/dockerfiles/misc:latest
  script:
    - docker buildx build -t ${CI_REGISTRY_IMAGE}:latest --build-arg VERSION=${CI_COMMIT_TAG} -t ${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG} --push --platform linux/amd64 --progress plain .
  rules:
    - if: $CI_COMMIT_TAG

This containerization script uses an image which includes docker and the buildx cli plugin. I often use the buildx plugin so that I can build for more than one platform/architecture and it also gives us a few extra niceties to use when it comes to experimental syntax.

If you wish to read up more about the buildx plugin, feel free to check out this post!

When the image is built, we tag it with the latest and tag of the current pipeline. The rules property only tells the gitlab ci scheduler that the job should only run on tags (where the $CI_COMMIT_TAG variable will be auto-populated by the engine to contain the current tag name).
The job have a needs clause defined so that all the artifacts from the jobs in the list are downloaded to the job!

Ofcourse, all of this requires a dockerfile, and the most simple one we could go with is a node image of some sort:

FROM jitesoft/node-base:latest
RUN mkdir /usr/local/app
ADD ./dist /usr/local/app/dist
ADD ./node_modules /usr/local/app/node_modules
ADD ./package.json /usr/local/app/

EXPOSE 3000
CMD ["/usr/local/bin/node", "/usr/local/app/"]

The image only copies the dist, node_modules and package.json files to the image and defines the CMD and exposes the port that is used by the application.
The image should be tested locally a couple of times, so that we haven’t messed something up, but if it runs as it should, it’s ready to be built!

The deployment!

What would a kubernetes deployment be without the actual deployment file?
That’s a good question, probably not much fun - would be my answer. So, at this point in the deployment process or rather in the process of writing a deploy script, we should run the initial deployment, that is, we want the pipeline to create an image that we can use when we deploy the application.
You could start off the deployment by just deploying another container instead, but I prefer to do a manual deploy with the real application as a first deploy, so that I know everything goes right and it works as intended.

So, the deployment would look something like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-awesome-app
  namespace: default
spec:
  selector:
    matchLabels:
      app: my-app
      env: production
      project: my-project
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    spec:
      containers:
        - name: my-node-container
          imagePullPolicy: Always
          image: registry.whereever.com/my-node-app:latest          
          resources:
            requests:
              cpu: 30m
              memory: 64Mi
            limits:
              cpu: 300m
              memory: 512Mi
          ports:
            - containerPort: 3000
              name: web

This deployment uses the image that we just created in the registry, it have limits and requests set for resources (something you should always do!) and it is placed in the correct namespace (this is important, as our deployer only have access to the default namespace).
As you might notice, we are using 3 replicas, and we only allow 1 to be unavailable at a time. This will create three identical pods with the same image and resources as defined in the template spec, so take all the requests and limits resources and multiply with three to see how much the cluster will require of a node to be able to schedule it successfully.

When you feel ready, you can deploy the deployment with kubectl:

kubectl apply -f my-deployment.yml

You will ofcourse require a few more resources than just the deployment to be able to access it, such as a Service and some type of ingress. But I’ll not get in to that here!

Deploy script

Now we have a deployment up and running, all that is left is to create the actual deploy script in the CI pipeline.

A really simple deploy script could look something like this:

.deploy:
  stage: deploy
  image: jitesoft/kubectl:latest
  variables:
    IMAGE: "registry.whereever.com/my-node-app:${$CI_COMMIT_TAG}"
  script:
    - kubectl set image deployment.apps/my-awesome-app my-node-container=${IMAGE}
  rules:
    - if: $CI_COMMIT_TAG

Just as with the containerization script in the earlier stage, we will only run this on tags.
The image used here is a standard kubectl image and the command we invoke is basically just a patch request of the deployment so that it changes the image to the latest build image.

In this stage, we really need to be able to access that kubeconfig file we created for the deployer earlier. So, you can either add it as a secret file, if the CI/CD engine allows for such (in that case, you will want an env variable to point to the file-path which is named KUBECONFIG), or you could add the whole file as a variable and then echo it into a file, which you then export a env variable for (KUBECONFIG=/path/to/config), just be sure to remove the file afterwards, so that no one else can use it!

When all is ready, just push the new CI file to the repository and add a tag to see how the deployment job runs!

Thats it! It’s a fully working awesome auto-deployment script! Every time you push a tag to the master branch, your new pipeline will kick in and test, build, containerize and finally deploy the image to your kubernetes cluster!

Final words

CI/CD and Kubernetes is a huge field, there are sooo much that could be done in so many ways, the one I describe in this post might not fit all, honestly, for some projects it doesn’t even fit me, but it is a simple enough (while still functional) deployment pipeline.

I hope you enjoyed the post!

Youtube - Me on Civo community meetup
Youtube - The whole Civo community meetup