Setting up your own Root Certificate Authority, aka Root CA, can be a difficult process. Web browsers and e-mail clients won't recognize your CA out-of-the-box, so most people opt to use public CA infrastructure. When security matters, using a public CA is the wrong solution. Privately owned and controlled CAs can be infinitely more secure than their public counterparts. However, most people who set up a private CA don't set up their CA infrastructure correctly. Here is what most private CAs look like:
Root CA cert -> Server cert
This is wrong because the server certificate has to be regenerated regularly (e.g. annually). If the root certificate is compromised, then it involves fairly significant effort to replace all of the certificates, including the root. What should be built is this:
Root CA cert -> Intermediate cert -> Server cert
In fact, this is the format that most public CAs use. The root CA cert is generated on a machine that isn't connected to any network. Then it is used to generate any necessary intermediate certs. Both the root and intermediates are signed for long periods of time - typically about 10-30 years for the root and 2-5 years for the intermediates. The root CA certificate private key is then physically secured - a physical vault of some sort helps here. The root CA is never, ever used on a network-connected machine. It is always used offline and it is only ever used to generate intermediate certificates. Only when specific conditions are met is the private key ever accessed. Usually those conditions entail a paper trail of accountability with multiple people who are physically present and are authorized to access the key for purposes declared in advance.
After the root CA generates the intermediate certificates and is secured, the intermediate certificates are then used to generate other certificates. The intermediates can possibly sit on a network connected machine, but that machine is behind a very well-guarded firewall.
So how does one create this setup, while implementing it easily and securely, AND relatively cheaply? First, you are going to need a few things:
Wall-powered USB hub with at least 4 open ports (you can't use built-in USB ports on a computer for this since they are usually underpowered - blame your motherboard manufacturer)
Six USB thumbdrives (ideally, brand new - they can have tiny storage and therefore be cheap - if you are doing this for personal use only, you can scale it back to three thumbdrives)
VirtualBox or VMware
KeePass
An ISO of a public Linux distro of some sort - smaller is better (e.g. Tiny Core Linux)
1 CD-R + CD burner
Labels and a pen or marker
Multiple, physically secure locations for storing the various thumbdrives
If you are familiar with virtual machines and constructing a root CA the right way, perhaps you can see where this is going. Hopefully, you will at least agree with me that where I'm going with this is a step above what most people do for their root CA and it is FAR easier and cheaper to isolate/secure a USB thumbdrive than it is a whole computer system.
Now let's get this started:
Make sure everyone is in the room who needs to sign off on the construction of the root CA and they are well-watered/coffee'd and have been to the restroom recently. They aren't going anywhere for a while. If someone leaves during this time, there's a pretty good chance the entire process will have to be restarted depending on how serious you are about the root CA. It will help speed things up considerably if the individual doing the technical side has gone through a dry run on their computer in advance to get familiar with the process. Also decide in advance which of the four people present will receive thumbdrives containing important data required to decrypt the virtual machine and the root CA private key.
Start an audit log file on the USB thumbdrive where the virtual machine will reside. Don't forget to label this thumbdrive with something like "Virtual machine, audit log". In the audit log, document the date, time, and notes for each operation - including each and every executed command against the virtual machine. The start of the audit log should declare who is present for the CA signing process as the first note. This file should be updated accordingly any time the thumbdrive is accessed in the future (e.g. to generate a new intermediate cert every couple of years).
The next step is to make the Linux distribution ISO read only to all users. Depending on the host OS, this step will vary. A simple solution to making the ISO file read only is to burn the ISO to a CD as an ordinary file (i.e. don't burn a bootable CD). If you plan to burn a CD, you might as well burn a copy of the verification information mentioned in the next step. This portion of the process can be done in advance but should be noted in the audit log that the ISO is confirmed to be read only.
Next, verify the ISO against public verification information such as a hash or PGP. Then compare the same public verification information from the same source at a secondary location in such a way that defends against a man-in-the-middle attack. A different computer or device on a completely different network satisfies this (e.g. a smartphone visiting the same URL over a cell network). If that isn't possible and the same machine as the one that retrieved the ISO is used, a secure session to a temporary VPS (e.g. a Digital Ocean droplet), 'wget' the URL containing the information, and then using 'cat' on the retrieved information may be sufficient for all present.
At this point, randomly generate the two main passwords for the virtual machine and CA root key using KeePass and store the database on the thumbdrive(s). If using a single thumbdrive for personal use, use the master password option. Otherwise, use the encryption key option and store the password database on one thumbdrive and the encryption key for the database on another. When operating in a group, clone each password database thumbdrive to another thumbdrive (up to four drives attached to the hub at this point - Hint: Plug in and remove drives one at a time to make labeling and cloning easier). Label each password database thumbdrive appropriately. They will be given to responsible individuals at the end of the process to keep secure along with instructions on use.
Now build the virtual machine with the ISO on the first thumbdrive, disabling unnecessary features such as audio. Enable encryption on the virtual machine disk image file and use the appropriate password from KeePass. Leave networking enabled for the moment. Install the OS, making sure that files are persistent across reboots. Install OpenSSL and verify that it runs. Then power off the virtual machine.
Disable networking. Physically disconnect the host's Ethernet cable/WiFi/etc. Disable/remove the optical drive controller in the virtual machine software. Connect the thumbdrive that will hold the certificates and keys to the host. Boot the virtual machine back up. Attach the newly connected thumbdrive to the guest using the virtual machine software (if it hasn't done so already) and then mount the thumbdrive inside the guest OS:
mkdir /mnt/thumbdrive
blkid
mount -t [TYPE] /dev/[something] /mnt/thumbdrive
Now you are ready to construct your root CA. From the terminal, run the following as root:
mkdir root_ca
chmod 710 root_ca
cd root_ca
openssl req -new -newkey rsa:4096 -x509 -days 3650 -keyout ca.key.pem -out ca.cert.pem
You will be asked a series of questions. Use the appropriate password from KeePass to protect the CA private key. Common Name should be something like "[Organization] Root Certificate Authority". Email Address should be left blank. 3650 is roughly 10 years. That's the length of time until you have to go through the whole process again. 7300 days is ~20 years, 10950 is ~30 years. Shorter times are, of course, more secure but create more hassle.
Then, run:
chmod 400 *.pem
openssl x509 -noout -text -in ca.cert.pem
cat ca.cert.pem
cp ca.cert.pem /mnt/thumbdrive/
The first command makes it so only the root user can read the files - why OpenSSL doesn't automatically chmod 400 everything that it generates is a security vulnerability that should be fixed. The second command dumps out the information about the certificate (be sure to verify that "CA:TRUE" appears). The 'cat' command dumps the raw PEM certificate data to the screen (a sanity check for PEM formatted data). The last command copies the signed certificate to the thumbdrive. If you ever accidentally dump or copy the 'ca.key.pem' file, then you will have to start over.
Now we are ready to generate an intermediate certificate which will be used to sign all other certificates. Run:
sed s/CA:FALSE/CA:TRUE/ < /etc/ssl/openssl.cnf > openssl.cnf
openssl req -config openssl.cnf -new -newkey rsa:4096 -days 1095 -keyout intermediate_01.enckey.pem -out intermediate_01.req.pem
chmod 400 *.pem
The first line alters the OpenSSL configuration so that you can generate an intermediate certificate that can be used to sign other certificates. Depending on the OS, the openssl.cnf file might not be at /etc/ssl/. The second command again asks a series of questions is asked. Similar sorts of responses should be used. Common Name should be something like "[Organization] Intermediate Certificate Authority". Leave the challenge password and company name empty. 1095 is about 3 years. That's the length of time until the virtual machine will have to be fired up again to generate a new intermediate certificate.
openssl x509 -req -days 1095 -in intermediate_01.req.pem -CA ca.cert.pem -CAkey ca.key.pem -extfile openssl.cnf -extensions v3_ca -set_serial 01 -out intermediate_01.cert.pem
chmod 400 *.pem
openssl x509 -noout -text -in intermediate_01.cert.pem
cat intermediate_01.cert.pem
cp intermediate_01.cert.pem /mnt/thumbdrive/
cp intermediate_01.enckey.pem /mnt/thumbdrive/
Finally, power down the virtual machine. Disconnect all thumbdrives. The thumbdrive labeled with the encrypted virtual machine and is moved into a physically secure location (e.g. a vault). The thumbdrive with the CA public key and intermediate CA files move over to other, properly firewalled, network-connected infrastructure. Once moved over, if automation is desired and authorized, the password can be removed from the intermediate CA private key:
openssl rsa -in intermediate_01.enckey.pem -out intermediate_01.key.pem
If you need more than one intermediate certificate or are renewing the certificate, adjust the above commands to increment all the 01's to the next available number (i.e. 02, 03, 04, 05, etc).
Four of the people present each receive one of the four thumbdrives containing the password database/decryption key after they sign for them. How this happens is up to the organization. Here is some sample verbiage: "I, ________, hereby accept one of the required components of the [Organization Name] root CA password data store. I will keep this device and the data on it in a secure location at all times. I will not copy, duplicate, or clone the data for any reason. I will not use the device or the data on the device on any machine connected to a network. If any of these terms are violated, disciplinary action may be taken against me up to and including termination. If I ever leave [Organization Name], I agree to transfer stewardship of this password data store to another [Organization Name] employee. Date/Signature"
This concludes how to correctly protect
Hey Thomas, so I read your post. So after I do all this, how do I install it on my VPS Linux server?
ReplyDeleteThanks, Corey
Okay. Let's assume that you make a root certificate and your first intermediate certificate as the start to the three-layer chain as per the article and video. From here you sign your server certificates with the intermediate certificate and private key. Since server software varies widely, there is no one-size-fits-all solution so I can't say "do this specific thing and it will work". However, the majority of software out there lets you configure it to use the server private key in one file and a "certificate chain" in another file containing: The root certificate, the intermediate certificate, and the server certificate. They could be in that order or in reverse order - it basically depends on the server software.
DeleteHere's what I do with all of my certificates on Linux servers for server software: I create a directory in /var called 'certs', so /var/certs. It is chmod 700 and owned by the root user. Then I generate a private key and CSR as one does. Then I get the CSR signed and put the resulting certificate into /var/certs. I clearly name each file so its purpose is clear. I build the certificate chain by 'cat'ing the root + intermediate + certificate. I also chmod 400 all of the files in /var/certs to make sure they are read-only and only the root user can see them. Most server software starts life running under root, so it has access to the files.
Of course, you still have to configure clients to accept your new root CA. You do that by installing the root certificate in the OS or browser trust store. Eventually we will use something like DNSSEC TSLA records, which would allow us to eliminate the root certificate stores in web browsers and OSes and finally secure our infrastructure.
Hope this helps you in your root CA endeavors!
Hello. Not sure if you will get this as I have just came across your post (it is fantastic by the way) but I am really struggling to figure out what goes on what USB stick. I am doing this for personal/hobby use. Please could you give me a clear bullet point of what goes on each of the 3 USB drives, if you don't mind, of course. Thank you very much in advance. Regards, Steve
ReplyDeleteThumbdrive 1: Audit log, KeePass portable, KeePass encryption keys (if any), and the virtual machine. Physically secure this somewhere. (Note that the KeePass encryption keys file in an enterprise setting store the file on a separate, physically secured thumbdrive.)
DeleteThumbdrive 2: Encrypted KeePass database. Physically secure this somewhere in a different location.
(Don't forget where the first two thumbdrives are stored or you will have to start over whenever the intermediate or root cert expires! You can't access the virtual machine without the passwords and you can't access the passwords without the password database. The password database is also useless without the decryption keys from the first thumbdrive if you use that option in KeePass.)
Thumbdrive 3: Root CA (public key), intermediate CA.
I prefer to use brand new thumbdrives from trusted manufacturers as USB thumbdrives can be compromised (e.g. malware installed). For personal use or for experimenting, you can get away with using some random thumbdrives lying around.
Thomas thanks for taking so much time to put this tutorial together! For those of is in "enterprise and business" who are new to your methodology of protecting the root CA with 6 flash drives and a CD, it may not be clear. Somewhere in the tutorial I thought you mentioned or eluded that others in the business will somehow take these other flash drives for safekeeping and have "parts" of what is required to activate the root CA.
DeleteThe explanation above is clear. But, what about the other scenario... 6 flash drives? What goes on each? Where does the CD go?
The CD contains the OS you will install into the VM. The CD is a throwaway part of the process. The ISO for the OS needs to be placed onto a read-only environment so that it can be validated as not having been modified in-transit. Some people might refer to this as a kind of "chain of custody." The Internet is deemed untrustworthy, so obtaining something from the Internet should be suspect. Validating the ISO with a secondary mechanism establishes trust that the ISO has not been modified from intended during or after the download prior to install.
DeleteFor 6 thumbdrives, the post walks through what to do, but it might not be completely clear. Here's the summary of what goes onto each thumbdrive and the destination for each:
1) Virtual machine and audit log. Goes into a physically secure location (e.g. a safe deposit box or a controlled-access vault).
2) Encryption key for the password database and KeePass Portable. Goes into a physically secure location that is different from where the first thumbdrive is located. It's important to keep track of both locations long-term. Putting the locations of all of the drives into the audit log is a good idea.
3-6) Copies of the encrypted password database. These are what are handed out. They are totally useless without the data on the other two thumbdrives above but you only need to have one of the four of these in order to access the virtual machine. People go on vacation, get sick, leave, etc.
This way, no one person can access the root CA but also deals with the scenario of accidentally losing a thumbdrive and thereby losing all access to the root CA and having to start over. You have to bring together three different pieces of hardware scattered among multiple locations and people and overcome physical controls to access the root CA. You can adjust the level of overkill as you see fit for your organization. Larger organizations with average turnover probably want all six. Smaller organizations might do just fine with four. The key to making it all work is establishing accountability and trust both in reality and digitally.
I also highly recommend using a security mechanism for long-term chain of custody of the thumbdrives themselves. In the video, I show anti-tamper money bags from MMF Industries. A few years ago, I spent several months coming up with a mechanism for transporting highly-sensitive material in chain of custody and tried pretty much every anti-tamper product on the market (and even delved into chemicals at one point). I was able to defeat nearly all anti-tamper mechanisms on the market with nothing but...patience. Seriously. Peeling back sticky material off of a surface (e.g. opening an envelope or raising an anti-tamper sticker) simply requires a few minutes of patient, slow peeling and it's virtually undetectable. And most anti-tamper tech on the market doesn't stop duplication of the encasing material. But it turns out that the banking industry solved the problem years ago. After having defeated pretty much everything on the market, I was doubtful they would hold up to my attacks as I'd gotten pretty good but I was completely unable to defeat the MMF Industries money handling bags. They're resistant to all types of attacks that I threw at it (thermal, physical, and chemical). If you want to transport or store something securely while maintaining chain of custody on the cheap, then those bags are a fantastic solution to the problem. They also happen to be an excellent solution to storing each thumbdrive for the root CA and later validating that the thumbdrive has not been accessed by anyone in the interim.
Hi Thomas,
ReplyDeleteExcellent instructions. You have obviously spent some time in a highly classified working environment. The term for a lot of what you presented, which I'm positive you are familiar with, is: "Air Gap".
I caught your youtube vid and followed it to your blogspot.
I setup a two tiered CA, no intermediary CA, for our VMware Horizon environment. I did this since we offer VMware IT Academy courses globally. www.cccti.edu/vmware
We use Horizon View as the portal into our virtual lab environment.
Oh, btw: I noticed you had VMware in you document as: "VMWare". They are somewhat particular on the way to present their name: either all lower case, or just: VMware.
After building the two tiered CA it became obvious that I need the intermediary CA.
I wanted to shorten the time for participants gaining access into our lab environment.
Not easy to do with with a two tiered CA.
I'm working on another project that you may find of interest, but I don't want to detract.
I have a tendency to chase rabbits!
I needed to revisit my CA deployment and thanks for doing your post which makes complete sense. I will be introducing the intermediary CA in the new design.
The one area of concern for me is the download of keepass from sourceforge since it does not have a checksum validation.
Not that significant of an issue. I'll just create my own password.
I'll update you on my progress, but it will be awhile, still have a few other fires burning.
Pete
Thanks for the tip about VMware. I've updated the post. I don't like it when people use CubicleSoft with a lowercase 's' but I understand why they do it.
DeleteWhoops. I didn't even think about double-verifying the KeePass download as part of the process. I already have it ready to go and use it as part of my workflow. KeePass also downloads over HTTPS while the OS portion comes down over plain ol' HTTP. So I was more concerned with verifying the OS, which would be housing the most critical components of the CA but you are definitely right, I should have verified KeePass too. I was also concerned with making a video of the process.
SourceForge does calculate a SHA-1 and MD5 hash if you alter the download URL by chopping off bits of the URL until it stops trying to download it. For example, the various versions of the latest 2.x release (at the time of this comment) can be seen here: https://sourceforge.net/projects/keepass/files/KeePass%202.x/2.38/
There I see a screen with all downloads for that version. There is a little 'i' icon next to each download. Each icon shows the calculated hashes.
The passwords for the virtual machine should never be known or shown. How secure you want your CA setup to be is up to you. Then again, I attempted to present the strictest environment and, even then, I missed the KeePass verification bit - actual security is hard!
I see a lot of two-tier CA setups and I'm pretty sure that is partly due to laziness, partly due to how the technology is built (e.g. having to alter config files to get an intermediate CA), and partly due to misunderstanding of how root CAs work by the broader public.
One technology to keep a close eye on is DNSSEC DANE TLSA. Under "Certificate usage 3", it is possible to run a private root CA and have it trusted by a remote client (e.g. a web browser would display the lock icon without having to install the root CA public key). All the more reason to figure out how to do a root CA the "right way" now and work out the bugs before a lot of people start building CAs for TLSA and set them up in a way that doesn't future-proof their environment (i.e. a two-tier vs. a three tier CA, not validating the chain of software used, etc).
Thomas,
DeleteThanks again for the rapid reply.
I woke up this morning with an additional thought.
btw: you are correct regarding laziness, but it can also be due to the lack of support/understanding from within the organization one is associated.
My thought/question: can the process be repeated to create an additional CA link?
Reason for this in my/our case is that for participants to gain access into our environment I need to have them install the CA cert.
I would like to have the "long-term" intermediary CA, and an additional "short-term" (3-month) CA.
Totally agree with your comment on securing the VM.
Were you ever familiar with another super small footprint linux called: dsl (damn small linux)?
I still have an iso, (for what it's worth?).
Thanks for the heads up, I'll investigate DNSSEC. Already on it.
fyi: I have developed a complete VMware Lab environment that can run using VMware Workstation. It does require (32G RAM) or (2x16G). It will fit on 256G SSD.
If you are referring to creating a 4-tier CA, the answer is yes. You can build chains however long you need them to be. The intermediate can be configured to sign another intermediate CA that can be used to generate server certificates.
DeleteHowever, you should be aware that a lot of software won't work too much beyond a 4-tier setup. The maximum allowed "certificate depth" is frequently hardcoded into the software itself, so changing it is usually impossible without recompiling the executable.
TinyCore was started by at least one of the core developers of DSL. That's part of why DSL kind of died off.
can you show v3_ca file . I do csr command and face error x509 unrecognised extension.
ReplyDeleteI wrote this post 4 years ago and each OS packages OpenSSL differently. So it is entirely likely that something changed in the interim OR there's something specific in your configuration file that the 'sed' line breaks. It's just attempting to replacing FALSE with TRUE on the CA line. It could be that it does something else for you or it's an entirely different problem. Not enough information to diagnose.
DeleteMy config file is buried on an air-gapped thumbdrive in an encrypted TinyCore virtual machine. ;)
Thomas, first of all, thank you for this in depth tutorial. I found it informative and very easy to follow. I noticed in your last reply that you are still monitoring this post.
DeleteMy question would be is there a follow up or update to any of these steps as technology has evolved and changed over the last 4 years or are the steps pretty much the same?
The steps haven't really changed much. I recommend reading the comments to get any clarifications as several people have asked really good questions (e.g. what each of the 6 USB thumbdrives are for in the 6 USB thumbdrive scenario).
DeleteThe most important thing to do is make sure you wind up with a three-tiered system: A Root CA that remains offline at all times, one or more Intermediate CAs that you put in trusted environments (could be network connected), and finally one or more certificates generated by the Intermediate CA. A two-tiered CA system is the wrong way because there's no way to revoke a Root CA if it leaks. The Root CA needs to be protected at all times and remain offline. This set of instructions guarantees that the CA is secure and that multiple people are required to work in concert in order to access it to digitally sign Intermediate CA certificates (a rare event).
The tools haven't really changed either. There are newer versions of various packages but the commands to execute are basically the same. You might want to consider using ECC private keys (i.e. elliptic curves) instead of RSA private keys but note that client application/browser/OS support for ECC is improving but most instructions out there still use RSA. There's nothing wrong with RSA keys per-se, but larger keys do take up more CPU resources during an initial TLS handshake.
For example, this line in the instructions:
openssl req -new -newkey rsa:4096 -x509 -days 3650 -keyout ca.key.pem -out ca.cert.pem
Could become two separate commands (one to generate the Root CA private key using ECC and the other to generate the CSR):
openssl ecparam -name secp384r1 -genkey -out ca.key.pem
openssl req -new -key ca.key.pem -x509 -days 3650 -out ca.cert.pem
Where "secp384r1" is the elliptic curve you might choose to use. There are several curves to choose from. RSA is easy by comparison: Toss the key generator a large number like 4096 bits and know the result will be pretty strong - albeit slower than a much smaller ECC key. ECC is relatively new compared to RSA, so the tools are slowly getting there to make working with them just as easy as RSA.
Hello Thomas,
ReplyDeleteThank you for posting exciting and really good video with instructions. Maybe I missed the link or detailed explanations - I wish I could understand the role of three- or six- pendrives in this endeavour. I do seriously consider following your instructions and that's the element I need to understande before taking "left" or "right" way for building my own, secure PKI using three-levels CA structure.
Thanks in advance for help.
Emil
The comments section above has additional details on this.
DeleteHi Thomas,
ReplyDeleteI followed all the steps. The certificates work fine on the computer (Mac OS) but on iOS it's not working. I get an infinite loop (lol) on Safari when trying to access any servers using the new certificates and on Chrome I get “NET::ERR_CERT_AUTHORITY_INVALID”. Any clues?
Thanks,
Américo
You need to add your root CA to the trust store of either the web browser or the OS or it won't be trusted and connection attempts will fail.
DeleteNote that Android will complain whenever the device boots when a private root CA certificate is installed.
Hello Thomas! I hope all is OK at your side after this long time since my first comment here.
ReplyDeleteI came back with question: how may I create LiveCD out of the ready-made, installed and working "tiny" OS together with some GUI for certificate management?
I know it is possible. Intent is to reach independence on virtualization platform, 10 years for root CA in this setup is OK, but what might happen to VM engines (any) over that time... ?
I wish I could kindly ask you for your "feasibility" comments. Obviously, the KeePass and certificates will still be stored on encrypted thumbdrives, but I hope to be able to boot this LiveCD on any machine (OK, nearly any).
Does it make sense? if yes - could you, please, share some tips.. ?
You could consider setting up a machine dedicated to managing the root certificate. SBCs, Mini PCs and laptops are relatively cheap these days. Or you could build a complete, working system (host OS + virtualization software) on an external drive. Might not boot very fast but it can be configured to never update its software. VM software rarely, if ever, makes breaking changes that causes it to stop being able to access VMs. I've imported VMs before that are fairly ancient and they start up just fine picking up where they left off.
Delete