Terraform and UpCloud

This article is part 1 in a series: Terraform, Ansible and UpCloud

• 1 - Terraform and UpCloud
• 2 - Terraform and UpCloud Part 2
• 3 - Ansible!

Running infrastructure on multiple providers is sometimes a pain. When I work on my “clouds” I try to keep then in a provider-agnostic way, that is, I try to make sure that I can provision servers on multiple providers and that they connect fine with my other servers in the cluster.
Even though I use many different providers, I do see one provider as my “primary” provider. That provider is UpCloud.

The main reason I like UpCloud is the fact that their support is very good. It’s easy to get ahold of someone if there are any issues (which I have to say have never happened during the 1-1½ years I’ve used them), they are easy to talk to and (from what I’ve heard) they are very quick with resolving any issues that arises.

I must note that I have only personally been in contact with the support about stuff that does not have to do with the acutal service, but rather with questions, requests and in some instances status updates.

But the support is not everything. Their prices and their very high-speed “Max-IOPS” disks is a very good reason to at the least try them out (I have all my rook-ceph nodes on UpCloud due to the speed of the disks).

Now, when I’ve said WHY I use them, I will give a small walk-through on HOW I use them!

to actually follow everything in this tutorial, you aught to have an account at UpCloud.

This post is a post in a series about Ansible, Terraform and UpCloud, but it might in some cases melt into my K8S series. A note will be added if so is the case! It’s also important to know that a lot of the stuff in the series is applicable to other providers just as well only the provisioning is strictly bound to UpCloud.

Terraform configuration files

When I last wrote about Terraform, it was still using HCL(v1) but the latest terraform versions are using HCL2. HCL2 have a lot of features that the earlier version did not.

Terraform is using a configuration language named HCL. HCL is in the bottom a superset of JSON, so you could basically build your terraform scripts using JSON. While it is possible, I do very much recommend using HCL instead of JSON, as it gives a whole lot more to the scripts.

HCL is quite basic, that is, it lacks some features that I would personally love, but it is possible to write plugins (from what I’ve seen, GO is the language of choice for this) and modules (which I will cover in a later post) for terraform, making it possible do stuff that the base-language does not support.
If you want to dive in to the documentation, you can find it at the terraform page.

All terraform scripts (.tf) in a directory will be used when terraform is invoked, they are parsed in alphabetic order.
While you can use a naming order to make the scripts run as you wish, you can also use depends_on in your resources to make resources depend on each other.

It’s customary to keep all variables inside its own file, I use a tf file for this (while you could use a .tf.json file if you prefer json (check the docs!)),

HCL is an abbreviation of HashiCorp configuration language, HCL2 is v2 of the language. HashiCorp have in a issue (link not yet re-located) said that terraform should/will be able to use both HCL and HCL2 and that HCL2 will be developed as its own language, that is, HCL2 will not be merged into HCL.

Defining a variable in the variable file is quite simple:

variable "variable_name" {
    type = "string"
    default = "This is a string variable."
}

Each variable declared like this is a global variable, it will be reachable from every config file via the var. prefix:

var.variable_name

There are other type of variables that you can define, but we save that for later and focus on a more “static” configuration for now.

When you write a terraform script, one of the main things you will use are something called a Resource. Each resource is a provisioned entity, such as a server, ip-address or a data object stored in the terraform state (I will try to cover this later too!).
Every resource uses a resource type which are provided by a Provider. There are a whole lot of providers, some are supported as standard terraform providers while some will have to be built (as the provider we will be using for upcloud).

When creating a new resource, the syntax used looks as the following:

reource "resource_type" "resource_name" {
}

The type and name creates a unique (per resource definition) identifier. Within the {} the resource data is added.
This does not mean that each new instance of a server will require a unique identifier, but rather each resource pattern.

To create multiple instances of a single type of resource, you can use the count resource variable:

resource "provider_name" "mini_server" {
    count = 1000
}

The above example will create 1000 instances of the mini_server resource. Maybe a bit too much, but yeah, you get the idea!

I will cover more parts of HCL later on, but for now, I think you get a gist of how it looks at the least.

Install terraform

Terraform is a single binary file. That is, if you don’t decide to build it from source, all it is is a executable file.
The easiest way to install it is to download the latest binary from the download page, find your operating system and architecture and download it as a compressed file.
Extract it and place the terraform binary in a suitable location.

On windows, you should probably place the terraform.exe file somewhere inside one of your Program Files directories, for example C:/Program Files/Terraform/terraform.exe, this path should also be added to the PATH environment variable to be able to reach terraform from the terminal you use.
On Linux-ish distributions, it can be extracted directly into any of the directories that will allow for usage of the program right away, I put mine in the /usr/local/bin directory, but anywhere you prefer should work just fine.

After that, terraform is “installed” on your computer and you should be able to verify that through terraform --version.

Install the UpCloud provider

The UpCloud provider is written in Go and the latest version of it requires version 0.12 of go. I will not go through the installation process of Go in this tutorial, but you an find information at the official page.

I have considered to provide pre-compiled binaries for the plugin, but have not had time to do it yet. This post will be updated if I do.

When you have go installed, the easiest way to get the source is to either download the latest release tarball or the full git repo (as a clone) from the terraform-provider-upcloud repository.
Compiling the provider (if you missed the information in the README file) is done by a few simple go commands:

go install
go build

When that is done, all you aught to do is to link or move the provider to the local users terraform plugin directory:

mkdir -p $HOME/.terraform.d/plugins
ln -s $(pwd)/terraform-provider-upcloud $HOME/.terraform.d/plugins
# alternatively:
mv $(pwd)/terraform-provider-upcloud $HOME/.terraform.d/plugins

Now the provider is installed and ready to be used!

Create the first resource

A provider is only supposed to be loaded once, if multiple times, terraform will throw an error and the scripts wont run. So I add my provider declarations inside my 01_variables.tf file:

provider "upcloud" {}

If you wish, you could set the login credentials of your API user in the upcloud provider clause, but I don’t, I prefer to let terraform ask me for the password and username instead, if you do too, just leave it as above.
It is also possible to provide the credentials via environment variables:

export UPCLOUD_USERNAME="api-username" 
export UPCLOUD_PASSWORD="api-password"

When we have declared the provider, we can start setting up resources. Our first resource will be a small server instance (the smallest there is on UpCloud!).

# The `upcloud_server` is the type that is defined in the 
# provider to let terraform know what kind of resource it is,
# in this case it is a "server" resource.
resource "upcloud_server" "my_first_resource" {
  count = 1
  zone  = "de-fra1" # This is the zone identifier on upcloud (this one is Frankfurt1)
  hostname = "my-first-resource" # the hostname will be the hostname of the server.
  plan = "1xCPU-1GB" 
  
  # The following declares the storage of the server,
  # the 1xCPU-1GB plan allows for a 25gb maxiops 
  storage_devices {
    tier    = "maxiops"
    size    = 25
    action  = "clone" # Clone tells upcloud to use an image that they provide.
    storage = "Ubuntu Server 18.04 LTS (Bionic Beaver)" # There are other OS'es, but this one is okay.
  }
  

login { # In this clause, we can define how to log in to the server.
          # Personally I hate using passwords, so we use a SSH key instead!
    user = "my-name"
    keys = [
        "my-public-ssh-key"
    ]
    create_password = false
    # If you wish to use passwords, set the above to true and add the following:
    # password_delivery = "sms" # which will send the generated password to your phone!
  }
}

# We can here define some output that we want terraform to print to the terminal when done
# in this case, it will print the public IP address of the server (check the upcloud docs to see all possible values)
output "server_ip" {
    value = upcloud_server.my_first_resource.ipv4_address
}

When the configuration script is created, enter the directory through the terminal and initialize terraform in the directory:

terraform init

When running the command a .terraform directory will be created in the directory. Inside this, some local terraform settings is stored.
Feel free to check it out, but don’t edit it if you don’t know what you are doing. If you accidentally do, just remove the directory and run init again to go back to start!
Init is an important command, because when you run it, terraform downloads or adds the plugins that is required to run the scripts.

Planning

When we have initialized the project, we should check so that everything is correct. This is done with the terraform plan command. When executing it, we should get some output which I will try to describe below…

If you decided to go the same approach as me when it comes to credentials, the first two things that will happen is that the UpCloud provider asks for a password and a username, this is the credentials for your API enabled UCloud user.

For safety, I would recommend creating a new user inside your account and give it access to the stuff that you wish it have, then enable the api access on only that user.

------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + upcloud_server.my_first_resource
      id:                                 <computed>
      cpu:                                "1"
      hostname:                           "my-first-resource"
      ipv4:                               "true"
      ipv4_address:                       <computed>
      ipv4_address_private:               <computed>
      ipv6:                               "true"
      ipv6_address:                       <computed>
      login.#:                            "1"
      login.2025795319.create_password:   "false"
      login.2025795319.keys.#:            "1"
      login.2025795319.keys.0:            "my-public-ssh-key"
      login.2025795319.password_delivery: "none"
      login.2025795319.user:              "my-name"
      mem:                                "1024"
      private_networking:                 "true"
      storage_devices.#:                  "1"
      storage_devices.0.action:           "clone"
      storage_devices.0.address:          <computed>
      storage_devices.0.id:               <computed>
      storage_devices.0.size:             "25"
      storage_devices.0.storage:          "Ubuntu Server 16.04 LTS (Xenial Xerus)"
      storage_devices.0.tier:             "maxiops"
      storage_devices.0.title:            <computed>
      title:                              <computed>
      zone:                               "de-fra1"


Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

This is the output from the terraform plan command. What it does is to describe what will happen when the apply command is executed. So from top:

Terraform tells us that + means create, that is, any resource which is prefixed with a + sign, will be created on the provider. The upcloud_server.my_first_resource is prefixed with a + and that is correct, it should create a new resource for us!
After the resource name we can see a bunch of set values, the id is set to <computed>, which is correct, that means that the upcloud api will create the server id for us, something that we can’t and really should not be able to control. The hostname was defined in the resource definition and is correctly set to my-first-resource, when the server is provisioned, it will have that hostname.
We told UpCloud that we wanted a public v4 IP in the resource, and due to not defining anything else, it also got a ipV6 address. We did not though define the actual ip to use, but rather let UpCloud assign one automatically (something that is not possible to do as of now).
After IP addresses we can see our first user. It’s set to not use passwords and the ssh key is set to a value which is expected to reside in the variables file.
We have defined a plan which is the smallest instance on UpCloud and the private network is enabled. The storage device is set to the values we defined in our resource declaration. The zone is “de-fra1” and that is also correct.

Now, if we want to provision the resource, we just run terraform apply.
What happens next is that terraform, through the UpCloud provider calls the UpCloud API. Through the API the resource (in this case a server) is created.
This takes about half a minute to complete, but when done, it will print the output we defined in the resource script, that is, the ipv4 address of the newly created server.

Open a terminal, write ssh my-name@<the ip from output> and you will be able to log on to the server that we just created!

If you wish to delete the resource, just type terraform destroy and ALL the resources you have defined in the configuration will be deleted (and in case of the UpCloud resources, removed).

Final words

As you might see, the terraform script does what we could have done through the UpCloud web interface. If you prefer it that way, go for it, but personally, I want as much as possible automated, that way the issues that comes in to play with the human factor is minimized; and that is a good thing in my humble opinion.

Terraform is a powerful tool, this post just scrape the topmost layer of it. Further on in the series you will see more advanced features of both terraform and HCL.