Friday, December 09, 2016

Setting up your own Root Certificate Authority - the right way!

Setting up your own Root Certificate Authority, aka Root CA, can be a difficult process. Web browsers and e-mail clients won't recognize your CA out-of-the-box, so most people opt to use public CA infrastructure. When security matters, using a public CA is the wrong solution. Privately owned and controlled CAs can be infinitely more secure than their public counterparts. However, most people who set up a private CA don't set up their CA infrastructure correctly. Here is what most private CAs look like:

Root CA cert -> Server cert

This is wrong because the server certificate has to be regenerated regularly (e.g. annually). If the root certificate is compromised, then it involves fairly significant effort to replace all of the certificates, including the root. What should be built is this:

Root CA cert -> Intermediate cert -> Server cert

In fact, this is the format that most public CAs use. The root CA cert is generated on a machine that isn't connected to any network. Then it is used to generate any necessary intermediate certs. Both the root and intermediates are signed for long periods of time - typically about 10-30 years for the root and 2-5 years for the intermediates. The root CA certificate private key is then physically secured - a physical vault of some sort helps here. The root CA is never, ever used on a network-connected machine. It is always used offline and it is only ever used to generate intermediate certificates. Only when specific conditions are met is the private key ever accessed. Usually those conditions entail a paper trail of accountability with multiple people who are physically present and are authorized to access the key for purposes declared in advance.

After the root CA generates the intermediate certificates and is secured, the intermediate certificates are then used to generate other certificates. The intermediates can possibly sit on a network connected machine, but that machine is behind a very well-guarded firewall.

So how does one create this setup, while implementing it easily and securely, AND relatively cheaply? First, you are going to need a few things:

Wall-powered USB hub with at least 4 open ports (you can't use built-in USB ports on a computer for this since they are usually underpowered - blame your motherboard manufacturer)
Six USB thumbdrives (ideally, brand new - they can have tiny storage and therefore be cheap - if you are doing this for personal use only, you can scale it back to three thumbdrives)
VirtualBox or VMWare
An ISO of a public Linux distro of some sort - smaller is better (e.g. Tiny Core Linux)
1 CD-R + CD burner
Labels and a pen or marker
Multiple, physically secure locations for storing the various thumbdrives

If you are familiar with virtual machines and constructing a root CA the right way, perhaps you can see where this is going. Hopefully, you will at least agree with me that where I'm going with this is a step above what most people do for their root CA and it is FAR easier and cheaper to isolate/secure a USB thumbdrive than it is a whole computer system.

Now let's get this started:

Make sure everyone is in the room who needs to sign off on the construction of the root CA and they are well-watered/coffee'd and have been to the restroom recently. They aren't going anywhere for a while. If someone leaves during this time, there's a pretty good chance the entire process will have to be restarted depending on how serious you are about the root CA. It will help speed things up considerably if the individual doing the technical side has gone through a dry run on their computer in advance to get familiar with the process. Also decide in advance which of the four people present will receive thumbdrives containing important data required to decrypt the virtual machine and the root CA private key.

Start an audit log file on the USB thumbdrive where the virtual machine will reside. Don't forget to label this thumbdrive with something like "Virtual machine, audit log". In the audit log, document the date, time, and notes for each operation - including each and every executed command against the virtual machine. The start of the audit log should declare who is present for the CA signing process as the first note. This file should be updated accordingly any time the thumbdrive is accessed in the future (e.g. to generate a new intermediate cert every couple of years).

The next step is to make the Linux distribution ISO read only to all users. Depending on the host OS, this step will vary. A simple solution to making the ISO file read only is to burn the ISO to a CD as an ordinary file (i.e. don't burn a bootable CD). If you plan to burn a CD, you might as well burn a copy of the verification information mentioned in the next step. This portion of the process can be done in advance but should be noted in the audit log that the ISO is confirmed to be read only.

Next, verify the ISO against public verification information such as a hash or PGP. Then compare the same public verification information from the same source at a secondary location in such a way that defends against a man-in-the-middle attack. A different computer or device on a completely different network satisfies this (e.g. a smartphone visiting the same URL over a cell network). If that isn't possible and the same machine as the one that retrieved the ISO is used, a secure session to a temporary VPS (e.g. a Digital Ocean droplet), 'wget' the URL containing the information, and then using 'cat' on the retrieved information may be sufficient for all present.

At this point, randomly generate the two main passwords for the virtual machine and CA root key using KeePass and store the database on the thumbdrive(s). If using a single thumbdrive for personal use, use the master password option. Otherwise, use the encryption key option and store the password database on one thumbdrive and the encryption key for the database on another. When operating in a group, clone each password database thumbdrive to another thumbdrive (up to four drives attached to the hub at this point - Hint: Plug in and remove drives one at a time to make labeling and cloning easier). Label each password database thumbdrive appropriately. They will be given to responsible individuals at the end of the process to keep secure along with instructions on use.

Now build the virtual machine with the ISO on the first thumbdrive, disabling unnecessary features such as audio. Enable encryption on the virtual machine disk image file and use the appropriate password from KeePass. Leave networking enabled for the moment. Install the OS, making sure that files are persistent across reboots. Install OpenSSL and verify that it runs. Then power off the virtual machine.

Disable networking. Physically disconnect the host's Ethernet cable/WiFi/etc. Disable/remove the optical drive controller in the virtual machine software. Connect the thumbdrive that will hold the certificates and keys to the host. Boot the virtual machine back up. Attach the newly connected thumbdrive to the guest using the virtual machine software (if it hasn't done so already) and then mount the thumbdrive inside the guest OS:

mkdir /mnt/thumbdrive
mount -t [TYPE] /dev/[something] /mnt/thumbdrive

Now you are ready to construct your root CA. From the terminal, run the following as root:

mkdir root_ca
chmod 710 root_ca
cd root_ca
openssl req -new -newkey rsa:4096 -x509 -days 3650 -keyout ca.key.pem -out ca.cert.pem

You will be asked a series of questions. Use the appropriate password from KeePass to protect the CA private key. Common Name should be something like "[Organization] Root Certificate Authority". Email Address should be left blank. 3650 is roughly 10 years. That's the length of time until you have to go through the whole process again. 7300 days is ~20 years, 10950 is ~30 years. Shorter times are, of course, more secure but create more hassle.

Then, run:

chmod 400 *.pem
openssl x509 -noout -text -in ca.cert.pem
cat ca.cert.pem
cp ca.cert.pem /mnt/thumbdrive/

The first command makes it so only the root user can read the files - why OpenSSL doesn't automatically chmod 400 everything that it generates is a security vulnerability that should be fixed. The second command dumps out the information about the certificate (be sure to verify that "CA:TRUE" appears). The 'cat' command dumps the raw PEM certificate data to the screen (a sanity check for PEM formatted data). The last command copies the signed certificate to the thumbdrive. If you ever accidentally dump or copy the 'ca.key.pem' file, then you will have to start over.

Now we are ready to generate an intermediate certificate which will be used to sign all other certificates. Run:

sed s/CA:FALSE/CA:TRUE/ < /etc/ssl/openssl.cnf > openssl.cnf
openssl req -config openssl.cnf -new -newkey rsa:4096 -days 1095 -keyout intermediate_01.enckey.pem -out intermediate_01.req.pem
chmod 400 *.pem

The first line alters the OpenSSL configuration so that you can generate an intermediate certificate that can be used to sign other certificates. Depending on the OS, the openssl.cnf file might not be at /etc/ssl/. The second command again asks a series of questions is asked. Similar sorts of responses should be used. Common Name should be something like "[Organization] Intermediate Certificate Authority". Leave the challenge password and company name empty. 1095 is about 3 years. That's the length of time until the virtual machine will have to be fired up again to generate a new intermediate certificate.

openssl x509 -req -days 1095 -in intermediate_01.req.pem -CA ca.cert.pem -CAkey ca.key.pem -extfile openssl.cnf -extensions v3_ca -set_serial 01 -out intermediate_01.cert.pem
chmod 400 *.pem
openssl x509 -noout -text -in intermediate_01.cert.pem
cat intermediate_01.cert.pem
cp intermediate_01.cert.pem /mnt/thumbdrive/
cp intermediate_01.enckey.pem /mnt/thumbdrive/

Finally, power down the virtual machine. Disconnect all thumbdrives. The thumbdrive labeled with the encrypted virtual machine and is moved into a physically secure location (e.g. a vault). The thumbdrive with the CA public key and intermediate CA files move over to other, properly firewalled, network-connected infrastructure. Once moved over, if automation is desired and authorized, the password can be removed from the intermediate CA private key:

openssl rsa -in intermediate_01.enckey.pem -out intermediate_01.key.pem

If you need more than one intermediate certificate or are renewing the certificate, adjust the above commands to increment all the 01's to the next available number (i.e. 02, 03, 04, 05, etc).

Four of the people present each receive one of the four thumbdrives containing the password database/decryption key after they sign for them. How this happens is up to the organization. Here is some sample verbiage: "I, ________, hereby accept one of the required components of the [Organization Name] root CA password data store. I will keep this device and the data on it in a secure location at all times. I will not copy, duplicate, or clone the data for any reason. I will not use the device or the data on the device on any machine connected to a network. If any of these terms are violated, disciplinary action may be taken against me up to and including termination. If I ever leave [Organization Name], I agree to transfer stewardship of this password data store to another [Organization Name] employee. Date/Signature"

This concludes how to correctly protect your nuclear arsenal and create a root CA on the cheap with OpenSSL.


  1. Hey Thomas, so I read your post. So after I do all this, how do I install it on my VPS Linux server?
    Thanks, Corey

    1. Okay. Let's assume that you make a root certificate and your first intermediate certificate as the start to the three-layer chain as per the article and video. From here you sign your server certificates with the intermediate certificate and private key. Since server software varies widely, there is no one-size-fits-all solution so I can't say "do this specific thing and it will work". However, the majority of software out there lets you configure it to use the server private key in one file and a "certificate chain" in another file containing: The root certificate, the intermediate certificate, and the server certificate. They could be in that order or in reverse order - it basically depends on the server software.

      Here's what I do with all of my certificates on Linux servers for server software: I create a directory in /var called 'certs', so /var/certs. It is chmod 700 and owned by the root user. Then I generate a private key and CSR as one does. Then I get the CSR signed and put the resulting certificate into /var/certs. I clearly name each file so its purpose is clear. I build the certificate chain by 'cat'ing the root + intermediate + certificate. I also chmod 400 all of the files in /var/certs to make sure they are read-only and only the root user can see them. Most server software starts life running under root, so it has access to the files.

      Of course, you still have to configure clients to accept your new root CA. You do that by installing the root certificate in the OS or browser trust store. Eventually we will use something like DNSSEC TSLA records, which would allow us to eliminate the root certificate stores in web browsers and OSes and finally secure our infrastructure.

      Hope this helps you in your root CA endeavors!