Certificate Authority with CFSSL
This post have been outdated and a new more up-to-date post can be found here!
Setting up your own CA
What is a CA?
A CA or Certificate Authority is the certificate that allows one to generate signed certificates to use during tls (transport layer security, earlier known as ssl, secure socket layer) communication.
If you visit a page with https, it’s using tls.
Tls allows for the communication to be encrypted and decrypted using secure keys and certificates. It’s quite important for the web and it allows us to set up services which can authorize by using TLS keys and certs.
Why do I want a CA?
A CA is not a necessity, but when building a K8S cluster, or when using other services which have the ability to use tls for authentication, a certificate authority is quite useful. Having your own allows you to use the same root authority for all your services, a thing that makes your whole infrastructure signed by your own authority. This is a good thing, but can of course create some issues if you sometime loose the CA root key. Always keep the root CA key in a secure location, if possible on a hardware that you can lock away. This is extremely important for larger organizations, as if their root CA is lost, their services will be open for quite nasty malicious attacks.
I keep my root ca on a tiny USB stick which I have locked away. All the root CA files are encrypted with gpg and signed with said keys, if someone locates the USB, they will still need to decrypt the files, which means that I must have lost both the USB AND the private key + password to my gpg key. I do not have the same security standard for my intermediate keys, but I try to keep those away from the net if possible!
Difference between the root and intermediate CA
The root CA is the most important part, it allows for generating intermediate certificates. Those can be used, just as the CA, to generate other intermediate certificates or to directly sign certs and keys. So, as a good standard, create one root ca, generate a set of intermediate certs and then hide the root ca key.
Each certificate which are generated with a intermediate ca will have references to the root, so the trust of the root will be handed down in the “trust chain”. The intermediate certs and the generated certs should be rotated from time to time, but I will not go over that here, but rather just describe the process of creating the chain of trust and the certificates that you might want to use for your page or for encrypted communication between services.
Generating
When generating certificates, we use a base program called OpenSSL. OpenSSL is a open source software which have been used a lot for many years. As with all security software, it’s a good idea to check out the CVE database - or as in this case, you can check out their official page and their vulnerability reports - before using it. The team behind OpenSSL are quite good with fixing their vulnerabilities, but one should always know that (as with all software) there can be bugs and there could be a CVE which makes the software less secure. If one is reported, you should most likely update the OpenSSL software ASAP.
Other than OpenSSL we can use a software created by Cloudflare, the software is a layer between you and OpenSSL which allows for creation of certs and authorities in a easier way. It also provides a web API which can be used (although I have not personally used it myself). The software in question is called cfssl and is free to use!
What we will do in this tutorial is to create a root certificate, a intermediate certificate and finally some certificates that could be used for a etcd service or something similar! We will also add the root to our trusted certificates so that we don’t have to get annoying “self signed cert” errors in the browser when using the certs.
Root cert and installation of CFSSL
As said before, the Root cert is the primary part in the chain of trust. The root is, as the name implies, the parent of all certs generated, each intermediate certificates are children of the root which are allowed to create their own child-certs.
By using a chain of trust, you have the ability to invalidate a intermediate certificate and all its children instead of having to re-do the whole certificate chain from top and down. In short, keeping a specific intermediate cert just for your infrastructure (or even go further, and use specific intermediate for each service) is a good idea.
So how do we generate a root certificate? As earlier mentioned, we will use a OpenSSL wrapper from Cloudflare called cfssl, this can be installed directly on your computer, or you can run it as a docker container (yay, no need to bloat the PATH var!).
I’ll focus on the local installation here.
Install locally
CFSSL requires a linux-like environment, i.e., linux, macos or wsl. I personally use windows as my main development environment, so I use WSL (ubuntu installation) for any linux-like environment and I’ll pretend that Ubuntu is the OS that you - the reader - use.
There are multiple ways to install CFSSL, but the absolutely easiest way is to just install it from apt:
apt get install openssl golang-cfssl
The package includes a few tools which are useful for creation and maintaining of certificates and the authority.
- cfssl - the command line utility
- multirootca - a multiple signing keys able certificate authority server
- mkbundle - a tool to bundle certificates
- cfssljson - a tool to extract certificates (key, crt etc) from the json output that cfssl produces
The two that we will focus on in this tutorial are the cfssl and cfssljson tools.
Authority
The initial certificate that we are to build is the root certificate, this is the one that you will want to keep safe later, and when I say safe, I mean really safe.
To keep it all as easy as possible we will use a directory structure so that we keep each authority cert in different directories and the actual certificates in a shared directory, each intermediate certificate will also have its own folder.
You do not have to follow this principle, I just prefer to have it structured like this to make it easy to look at!
/root
/ca
/intermediate
/cluster
/development
/certs
The root certificate will be in the ca directory, and we will name it ca.*.
To create the CA we need to create a signing request. This is done with JSON and mainly contains information about the authority owner. As an example, I’ll show the one I use for my own CA:
{
"CN": "Jitesoft CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "SE",
"L": "Lund",
"O": "Jitesoft",
"ST": "Skania"
}
]
}
The CN (Common Name) key is the authority name, in my case, I use my company name and append CA just to make it clear what it is, choose a name that you feel is good for your authority. The key object contains information about how the key is to be signed.
The names array contains an object, the short codes means: C ISO 3166-2 country code (SE = Sweden), L Locality, I.E., city name of the company, O Organization, in my case ‘Jitesoft’ as that is my company name, ST State. You could also add OU, which stands for Organizational Unit, to this list if you wish, but in this case, I left it out, as it is the root of all the certs!
When the file is created, you can save it as ca-sr.json (ca sign request) in the ca directory and open the terminal there.
Now we generate the first of our certificates:
cfssl gencert -initca ca-sr.json | cfssljson -bare ca
The above command creates a new certificate, a key and a sign request which can be used to cross-sign with other CAs. When these are created, cfssl will produce some json output which we pipe to the cfssljson tool, telling it to create files from the result. The files will be named by the last argument (which in the above case is ca). So now we will have the following files in the ca directory:
ca-key.pem(certificate key)ca.pem(certificate)ca.csr(sign request)
The ca.pem file is a public file, you can upload this to your facebook if you so wish (well, that might not be too useful…), the ca.csr file is not too important as of now, as we won’t sign the request with another CA, while the ca-key.pem is your private key.
This key is the object that you should keep safe. Keep it as safe as possible after you are done with this tutorial and do not share it with anyone. If anyone gains access to the CA, they can sign requests as if it was you doing it.
Intermediate
We now have our certificate authority up and running! Great! Next on the agenda is to create a few intermediate certificates that can be used for different services or areas.
The intermediate directory contains cluster and development directories, I thought those two could be a good base for the tutorial. The cluster authority will be designated to manage the cluster that you (will!!) run in the future, while the development authority could be used for development certificates, something that might not be too security heavy, but still good to have to be able to use TLS certificates which your development machines trusts.
Just as with the root, each intermediate certificate requires a signing request json file. If you are super lazy, the one I used above could be used for all, but if you want to set a different name or use different “names” (maybe even specify that OU key!), you could copy the file and put it in the folder for the specific intermediate cert.
Each intermediate should have its own usages, not all certs needs to be able to do whatever, but should rather just have the ability to create or sign certificates depending on their usage. A certificate intended to be used to sign emails should probably not be able to create new keys or sign code. So we specify only the key usages that we feel is needed.
When creating the configuration, we use a single file for all the configurations, when doing this, we set up a new “profile” for each of our intermediate certificates.
We create the file: /intermediate/config.json
{
"signing": {
"default": {
"expiry": "43800h"
},
"profiles": {
"cluster": {
"usages": [
"signing",
"key encipherment",
"cert sign",
"crl sign",
"server auth",
"client auth"
],
"ca_constraint": {
"is_ca": true
}
},
"development": {
"usages": [
"signing",
"key encipherment",
"cert sign",
"crl sign"
],
"ca_constraint": {
"is_ca": true
}
}
}
}
The above configuration contains our two profiles development and cluster. Each are given a set of usages and a object called ca_constraints in which we mark the certificate as a CA.
The following commands will generate and sign your intermediate certificates:
(intermediate.json is the sign request which is basically the same as for the root ca while the config.json is the configuration above).
cd intermediate
cd development
cfssl genkey -initca ../intermediate.json | cfssljson -bare development
cfssl sign -ca ../../ca/ca.pem -ca-key ../../ca/ca-key.pem --config ../config.json -profile development development.csr | cfssljson -bare development
cd ../cluster
cfssl genkey -initca ../intermediate.json | cfssljson -bare cluster
cfssl sign -ca ../../ca/ca.pem -ca-key ../../ca/ca-key.pem --config ../config.json -profile cluster cluster.csr | cfssljson -bare cluster
By now the cluster and development directories should have been filled with each of its own *.pem, *-key.pem, *.csr files.
With a intermediate certificate you are able to create other intermediate certificates, if you wish to do that, do the same as above but use the wanted intermediate cert as the certificate that you sign the requests with. This could be a good idea to do if you wish to have a bit higher security when it comes to the different services of your cluster, or different parts of your infrastructure. But you don’t have to, and I won’t go through it more in this post.
Now when we have our intermediate certs, we can encrypt, zip, encrypt and then encrypt our ca key file, we can then encrypt it a few times more, create a image from the charcodes of the encrypted file and print it on a paper, we then split the paper in 8 pieces and put them in lockers at 8 different banks, write down the banks on a paper, encrypt it and zip and encrypt and print and add to a 9th bank box. By then we have a semi-secure location that our CA key is stored, might be a bit annoying if you ever need the key again, but at the least any possible thief will have to deal with a lot of annoying steps to get it!
Well, we might not have to go to that length to secure it, but make sure that it is not easy to come over and remove the key from the local computer!
Certificates
Now when we intend to create certificates, we aught to use the correct intermediate for the different certificates that we want.
We will in this part focus on the cluster certificates and we will set up a few certificates that our cluster should be able to use when communicating through TLS.
Just as with other certificates, our new certificate will require a signing request and a configuration file. This time though, we will specify profiles for three different type of services:
server, peer and client. The server certificate should be able to sign, encrypt, decrypt and authenticate the other certificates, the server certificate is used by the server services that expect connections over TLS. The peer certificates are used for the services (that uses the server certificate) to communicate with eachother, while the client certificate is used by any service that needs to be able to communicate with the servers.
We could, if we where really lazy just create a certificate that could do all of this, but that would make security a bit worse, so we should not!
The configuration will look something like this:
{
"signing": {
"default": {
"expiry": "43800h"
},
"profiles": {
"server": {
"usages": [
"signing",
"digital signing",
"key encipherment",
"server auth"
]
},
"peer": {
"usages": [
"signing",
"digital signature",
"key encipherment",
"client auth",
"server auth"
]
},
"client": {
"usages": [
"signing",
"digital signature",
"key encipherment",
"client auth"
]
}
}
}
}
In the above configuration, the peer certificate will be the one with the most authority, but that is expected, as the peers will need to authenticate in both directions. The server certificate will be able to authenticate as server, while client only as client.
When we generate certificates, we will have to create singing requests for each of the profiles, they are a bit different from the earlier certificates and should look something like this:
server.json
{
"CN": "Server",
"hosts": [
"127.0.0.1",
"server.domain",
"sub.domain.tld"
]
}
As you see in the server certificate, it contains a list of hosts , the hosts should be the IP or FQDN that the servers uses. The peer certificate should basically use the same, as the servers and peers will be used by the same machines.
peer.json
{
"CN": "Peer",
"hosts": [
"127.0.0.1",
"server.domain",
"sub.domain.tld"
]
}
When it comes to the client certificate, we don’t set any hosts, as we want to be able to connect to the services without having to care about the clients current host. It’s okay to do though, if you want to!
client.json
{
"CN": "Client",
"hosts": [""]
}
It’s possible to create wildcard certificates by using the * character, but it’s seen as a security risk.
When we got all the files ready, we just create the new certificates with the help of CFSSL:
cfssl gencert -ca=intermediate/cluster/cluster.pem -ca-key=intermediate/cluster/cluster-key.pem -config=config.json -profile=client client.json | cfssljson -bare client
cfssl gencert -ca=intermediate/cluster/cluster.pem -ca-key=intermediate/cluster/cluster-key.pem -config=config.json -profile=peer peer.json | cfssljson -bare peer
cfssl gencert -ca=intermediate/cluster/cluster.pem -ca-key=intermediate/cluster/cluster-key.pem -config=config.json -profile=server server.json | cfssljson -bare server
By now, we should have 3 files for each of our configurations:
client-key.pem
client.pem
client.csr
peer-key.pem
peer.pem
peer.csr
server-key.pem
server.pem
server.csr
Put them in the correct directories where you wish to keep them!
I recommend that you generate new certificates for every new service, that way you can easily rotate the certificate if it is compromised.
When this is done, you should verify the keys with OpenSSL:
openssl x509 -in <cert>.pem -text -noout
Remember to store the private key at a secure location!
Adding your CA to the computers certificate storage
When you have certificates, you might want to allow your computer (and servers) to let the certificates to be seen as secure (that is, no annoying ‘SELF SIGNED CERTIFICATE ERROR!’). Depending on your OS, this is done in a few different ways.
Each computer have a trusted store, and to make our certificates seen as secure on a specific computer, we need to add the public certificate to the trusted store. We don’t add each of the certificates to it, as that would be very annoying, but instead we add the root certificate to the trusted store, that way all the certificates created in the same trust-chain will be seen as secure by the computer in question.
On windows, this is done through a certificate utility which you can find if you search on “Manage Computer Certificates”, once inside the utility, open the Trusted Root Certificates and right click on certificates and import the certificate public key.
On debian and ubuntu based machines, you do it easiest by installing the ca-certificates package, then copy the public key to the /usr/share/ca-certificates directory and then run the update-ca-certificates command (likely as super user or via sudo).
Now your certificates will be seen as secure by the computer!
If your computer requires the certificate in der / .crt format, use OpenSSL to convert the certificate with openssl x509 -outform der -in certificate.pem -out certificate.crt.
Last words
Setting up a CA is not that hard and is a very important part of your infrastructure and security. There is a lot more to certificates than what I went through here, but hopefully this will allow you to get a general look on how it works. This post will be referenced in the Kubernetes series I write, as we will use certificates created from our own authority for all services there.
As always, let me know if anything is wrong or if I missed anything.