HA k3s - Deploying the network

This article is part 3 in a series: HA K3s with terraform and ansible

• 1 - HA k3s provisioning
• 2 - HA k3s - Networking in Hetzner
• 3 - HA k3s - Deploying the network

As stated in the first post in this series, my plan is to deploy the cluster with the help of CI/CD pipelines in gitlab.
Even if you don’t plan to deploy from a pipeline, some parts of this post could be useful still, as I will write about states, and especially remote states.

I’ll try to use the TF/tf abbreviation for terraform and opentofu, as they both use that themselves. The projects are still quite close to each other in functionality, but when it comes to the GitLab implementation, it’s OpenTofu that is used.

If you use Terraform, the encryption part of this guide might not work as intended and you might want to leave any auto encryption out. The encryption clause is new to OpenTofu.

TF in GitLab

GitLab have quite good support for Terraform and OpenTofu both as a state storage as well as a module registry, further, they supply ci/cd components to ease setting up deployment of the projects, which is really nice.

TF state

When you deploy a tf project, a state is created, the state describes the currently deployed infrastructure.
To allow sharing of states (excluding sending files), one of the easiest and best ways are to use a remote state storage.

There are a bunch of storage types, but in this case, I decided to use the one that GitLab provides (as that’s where I have my files and pipelines).

To tell gitlab to use their storage as your remote state storage, you aught to add a new object to the terraform clause.
In my case, I added a new terraform file named backend.tf which initially looks like this:

terraform {
  backend "http" {}
}

On the tf side, that is basically all you have to do for gitlab to create the state in their storage.
Now, that does not mean that we are done, we need a pipeline!

As I earlier mentioned, gitlab provides a set of components, and to make it super easy, we will use the full-pipeline version.

The .gitlab-ci.yml file should look like this:

include:
  - component: $CI_SERVER_FQDN/components/opentofu/[email protected]
    inputs:
      version: 0.50.0
      opentofu_version: 1.9.0
      auto_encryption: true
      auto_encryption_passphrase: $ENCRYPTION_PASSPHRASE

stages: [validate, build, deploy, cleanup]

The components are versioned, so I would expect the 0.50.0 component to keep on working, but I would recommend checking the Component repository and see if anything new have been added.

The above pipeline will run the following jobs:

Stage validate:

• fmt (will run a tofu format command)
• validate (will run a tofu validate command)

Stage build:

• plan (will run a tofu plan command)

Stage deploy:

• apply (will run a tofu apply command)

Stage cleanup:

• destroy (will run a tofu destroy command and tear down the infra)
• clean-state (will delete the state stored in the remote state storage)

The Deploy and Cleanup stages are both ‘Manual’, which means that you will have to actively invoke them in the GitLab UI for them to run (which is quite good, seeing you don’t want to accidentally delete your infrastructure!).

As you can see in the pipeline file, there are a few inputs which are set:

version is the component version, from my understanding it is used for the sub-components, and should for now be set, while there is an issue in the gitlab tracker to make it use the value from the initial component inclusion string.

opentofu_version is the version of the OpenTofu executable. As of writing this, 1.9.0 is the latest, but to make sure the version you choose is correct, take a look in the component repository.

Now to the important stuff…

auto_encryption is a boolean value which tells gitlab to include a TF_ENCRYPT variable, which in turn activates automatic encryption of the state file.
This is likely something you will want to do, but it’s important to know that you need to save your encryption passphrase somewhere safe, so that you can decrypt the state in case something goes wrong.

auto_encryption_passphrase is the passphrase used to encrypt the state, I use a GitLab variable which is protected, masked and hidden.

Deploy the first version

Seeing we use some custom provider passwords and such, we need to create a tfvars file that the deployment can use, a tfvars file is a simple key-value file which we define variable values in (seeing gitlab isn’t interactive!).

The file should look something like this:

hcloud_token = "my-hcloud-token-created-in-previous-post"

Now, we don’t really need to save this file anywhere, but as we will want a variable file later on, we might as well create a dev.tfvars file in the root of the project and add it to .gitignore.

The text in the file should be added to gitlab though.
We don’t want to commit a secret, so for this, we use the Variables and mark it as a file.

Name the Key to GITLAB_TOFU_VAR_FILE and gitlab will pick it up in the commands!

When this is done, we can push the repository to the main branch and check the pipelines…

Press apply and your network will be deployed to Hetzner!

State and Validate…

Now, the component we use currently (as of 20250113) have an issue…
When the state file is encrypted and the validate command runs, it will not use the backend at all, this means that it can’t decrypt the state, hence not download the Hetzner provider.

I have yet found a good way to get around this, so in my CI pipeline, I added a rule to never run the validate job:

include:
  - component: $CI_SERVER_FQDN/components/opentofu/[email protected]
    inputs:
      # ... 
      validate_rules:
        - when: never

With this change, the validate job won’t run and no failure will happen.

Access the state locally

When we work with tf, it’s quite often that we want to make sure that the plan is possible to apply
(As you probably know, this is done with the terraform|tofu plan command).
To plan locally, you need the state. But now our state is encrypted and placed in a remote location…

To allow the state to be downloaded and decrypted in your local environment, there are two things that have to be done:

Encryption in the terraform backend

In our backend.tf file, we need to add a new clause called encryption.

terraform {
  backend "http" {}

  encryption {
    key_provider "pbkdf2" "gitlab_tofu_auto_encryption" {
      passphrase = var.gitlab_state_encryption_passphrase
    }

    method "aes_gcm" "gitlab_tofu_auto_encryption" {
      keys = key_provider.pbkdf2.gitlab_tofu_auto_encryption
    }

    state {
      enforced = true
      method   = method.aes_gcm.gitlab_tofu_auto_encryption
    }

    plan {
      enforced = true
      method   = method.aes_gcm.gitlab_tofu_auto_encryption
    }
  }
}

We define the key_provider, which is a pbkdf2 (a PSK crypto function) and the same as the one GitLab currently uses, it should basically just set the passphrase property to the passphrase value.
In my case, I added the passphrase to my local tfvars file and the variables.tf file I use to expose it.

The next part is the method used to decrypt the state, this uses aes_gcm, which is an AES based gcm algorithm, the same as the one GitLab currently uses, the keys parameter is set to the key_provider we created above.

We then set the state and plan objects to enforce encryption and to use the method we defined.

Initialize tf with backend config

When this is done, we need to init the tf project with a lengthy command, the full command can be found in gitlab under the Operate > Terraform States tab, press the vertical ... under Actions and select the Copy Terraform init command.

The command looks like this:

export GITLAB_PROJECT_ID=<PROJECT-ID>
export GITLAB_ACCESS_TOKEN=<YOUR-ACCESS-TOKEN>
export TF_STATE_NAME=default
tofu init \
    -backend-config="address=https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/terraform/state/$TF_STATE_NAME" \
    -backend-config="lock_address=https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/terraform/state/$TF_STATE_NAME/lock" \
    -backend-config="unlock_address=https://gitlab.com/api/v4/projects/${GITLAB_PROJECT_ID}/terraform/state/$TF_STATE_NAME/lock" \
    -backend-config="username=<your-username>" \
    -backend-config="password=$GITLAB_ACCESS_TOKEN" \
    -backend-config="lock_method=POST" \
    -backend-config="unlock_method=DELETE" \
    -backend-config="retry_wait_min=5"

After this have been invoked, your local state will be updated from the remote state, now we can try tofu plan and see that the state is downloaded and decrypted.

Final words

And that’s it. We now have a way to deploy our network to hetzner with a CI/CD pipeline! Kinda nice and easy eh?

There are additional things that can be (and I will try to write about here) done with the pipeline, one thing I really want to add to my own pipeline is to do a tofu plan on pull requests and display the infra difference in the PR directly.

As usual, let me know if you find any oddities in the post and I’ll correct it asap!