Smart Card/HSM backed OpenSSL CA

This article describes how to set up a Smart Card/HSM backed OpenSSL CA using a Smart Card HSM or any PKCS11 enabled device.

Background

Since some years back I use WPA2 Enterprise with EAP-TLS (Certificate authentication) for my wifi at home. Historically I have used certificates from a public CA for this purpose. This is not best practice since you don’t have control over the certificates that are issued.

Also, I recently bought a new switch capable of 802.1X authentication on all ports. For this purpose I want all my machines (even those without wifi) to have certificates. So I decided to go through the hassle of setting up my own private CA.

Setting up CA

For the basic setup of the CA I followed Jamies excellent guide on setting up a CA. So in this post you can assume that all the basic stuff like folders structure and basic commands are the same. I will only show you the differences needed to have the Root CA key stored on a PKCS11 device like a HSM, Smart Card HSM or a Yubikey. I will even try to follow his topic names so you can follow along.

Configure PKCS11 Engine

I will not discuss the operating system part of getting PKCS11 devices to work in this article. But basically you just need to install some packages, you can read about it here.

First of all we need to configure OpenSSL to talk to your PKCS11 device. This can be done from configuration or interactively on the command line.

From conf:

# At beginning of conf (before everything else)
openssl_conf            = openssl_def

# At end of conf (after everything else)
[openssl_def]
engines = engine_section

[engine_section]
pkcs11 = pkcs11_section

[pkcs11_section]
engine_id = pkcs11
dynamic_path = /usr/local/lib/engines/pkcs11.so
MODULE_PATH = /usr/local/lib/opensc-pkcs11.so
init = 0

From cli:

OpenSSL> engine -t dynamic -pre SO_PATH:/usr/local/lib/engines/pkcs11.so -pre ID:pkcs11 -pre LIST_ADD:1 -pre LOAD -pre MODULE_PATH:/usr/local/lib/opensc-
pkcs11.so

Create the root pair

First of all we need to have a RSA key pair on the PKCS11 device:

# pkcs11-tool --module /usr/local/lib/opensc-pkcs11.so -l --keypairgen --key-type rsa:2048 --label "SSL Root CA"
Using slot 0 with a present token (0x0)
Logging in to "HSM 2 (UserPIN)".
Please enter User PIN:
Key pair generated:
Private Key Object; RSA
  label:      SSL Root CA
  ID:         d15c3e9578a612a658bb14e0e147db4f2279cf19
  Usage:      decrypt, sign, unwrap
Public Key Object; RSA 2048 bits
  label:      SSL Root CA
  ID:         d15c3e9578a612a658bb14e0e147db4f2279cf19
  Usage:      encrypt, verify, wrap

Create the root certificate

I will assume that you have configured pkcs11 in openssl.cnf (otherwise you will have to first run the engine command in openssl interactively before any other command).

# openssl req -config openssl.cnf -new -x509 -days 7300 -sha256 -extensions v3_ca -engine pkcs11 -keyform engine -key 0:d15c3e9578a612a658bb14e0e147db4f2279cf19 -out certs/ca.cert.pem
engine "pkcs11" set.
Enter PKCS#11 token PIN for HSM 2 (UserPIN):
0x8018b6000 07:41:35.523 cannot lock memory, sensitive data may be paged to disk
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [SE]:
State or Province Name []:
Locality Name []:
Organization Name [PeanOrg]:
Organizational Unit Name []:PeanOrg Certificate Authority
Common Name []:PeanOrg Root CA
Email Address []:

Create the intermediate pair

For the intermediate key pair I followed jamies guide. I need frequent access to this CA so I have decided to have the intermediate pair on file instead of HSM.

Create the intermediate certificate

I changed one thing in jamies intermediate/openssl.cnf because I dont see the point of having province set in the CAs

stateOrProvinceName     = optional

To use the Root key stored on pkcs11 to sign the intermediate certificate use this command:

# openssl ca -config openssl.cnf -extensions v3_intermediate_ca -days 3650 -notext -md sha256 -engine pkcs11 -keyform engine -keyfile 0:d15c3e9578a612a658bb14e0e147db4f2279cf19 -in intermediate/csr/intermediate.csr.pem -out intermediate/certs/intermediate.cert.pem
Using configuration from openssl.cnf
engine "pkcs11" set.
Enter PKCS#11 token PIN for HSM 2 (UserPIN):
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4096 (0x1000)
        Validity
            Not Before: Apr  7 05:54:22 2018 GMT
            Not After : Apr  4 05:54:22 2028 GMT
        Subject:
            countryName               = SE
            organizationName          = PeanOrg
            organizationalUnitName    = PeanOrg Certificate Authority
            commonName                = PeanOrg Intermediate CA 1
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                77:9C:07:23:FD:40:E9:5C:7E:30:73:8F:59:28:25:F5:06:43:B4:70
            X509v3 Authority Key Identifier:
                keyid:A4:F2:DE:15:8E:9E:A8:87:B0:95:D4:21:A2:BD:4C:41:02:93:E0:8D

            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
Certificate is to be certified until Apr  4 05:54:22 2028 GMT (3650 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

We now have all we need to sign certificates. Just follow Jamies guide Sign server and client certificates

References

It took me a few hours to get this going because of sort of a lack of documentation on how to use OpenSSL and PKCS11 together, during my efforts I found these resources helpful

Strong authentication of OpenSSH hosts using hardware tokens

This article describes how to secure your SSH host keys using hardware tokens.

Background

I have previously written numerous posts about strong user authentication using smart cards, yubikeys and/or OpenSSH certificates. I have also described how you can use OpenSSH certificates to authentcate hosts. But let say your server where compromised in some way just for a few hours and your SSH host keys (and certificates) where stolen. This would allow the attacker to perform MITM attacks or impersonate your server in several different ways. Depending on the application this could be more or less detrimental. If the server have a lot of users this could be used to steal passwords for example.

One way to mitigate this attack is to store your host keys on hardware tokens. This way they cannot be stolen by someone not in physical contact with the server, and you would easily find out if the token was missing.

Configure the token

I will use a Yubikey 4 for this. I have already described how to use yubikeys for client keys, so a more detailed description on how to configure and use Yubikeys for SSH can be found here. You can check that everyting is working like this:

# opensc-tool -l
# Detected readers (pcsc)
Nr.  Card  Features  Name
0    Yes             Yubico Yubikey 4 OTP+U2F+CCID 00 00

What we need is basically a pkcs11 capable device i.e a smart card or “similar” and create a  RSA key pair on it.

Configure sshd and setting up a ssh-agent

Since it would be extremely impractical to enter the token PIN every time a client connects to your secure server, we will need to use a ssh-agent that will keep the pin in memory as long as the agent process is running. We also need to enter the PIN before we leave:

# ssh-agent -a /root/yubikey-agent
setenv SSH_AUTH_SOCK /root/yubikey-agent;
setenv SSH_AGENT_PID 10894;
echo Agent pid 10894;
# setenv SSH_AUTH_SOCK /root/yubikey-agent;
# ssh-add -s /usr/local/lib/opensc-pkcs11.so
Enter passphrase for PKCS#11: 
Card added: /usr/local/lib/opensc-pkcs11.so
# ssh-add -l
2048 SHA256:qBbMpdbUeabLe4PnfjrjPbGPu8zfbkbK+ni4mXOnV24 /usr/local/lib/opensc-pkcs11.so (RSA)

Now you have your RSA key available to the system using the UNIX-domain socket at /root/yubikey-agent.

In a production situation I would put this in a rc-script and enter the PIN at boot-time.

Configure OpenSSH

We need to  create a file containing the public key and also tell sshd to use this key and the newly created socket as a backend for its host keys.

# ssh-keygen -D /usr/local/lib/opensc-pkcs11.so > /etc/ssh/yubikey_host_key.pub
# echo "HostKey /etc/ssh/yubikey_host_key.pub" >> /etc/ssh/sshd_config
# echo "HostKeyAgent /root/yubikey-agent" >> /etc/ssh/sshd_config

Notice that if you have several keys on your token, ssh-keygen will output all of them. Make sure only the correct keys are added to the key file.

Verify the configuration

When you have configured sshd you will need to restart sshd and them we can verify that host keys are actually from the hardware token (in this case the yubikey). A very easy way to do this is to actually try to connect to the server and have a look at what keys it presents to you.

% ssh server
The authenticity of host 'server (172.25.0.15)' can't be established.
RSA key fingerprint is SHA256:qBbMpdbUeabLe4PnfjrjPbGPu8zfbkbK+ni4mXOnV24.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)?

You can now easily verify that this fingerprint IS actually the same as the one ge got when we added the yubikey to the ssh-agent.

This is basically it. If you are going to use this in production you would probably want to add some rc-scripts to automate the setup process as much as possible.

SSH certificates together with hardware keys (smartcard/yubikey)

We have showed how to use SSH certificates and SSH CAs, we have also showed how you can use the yubikey to store you SSH keys. This article will describe how to combine these two features.

First of all you need to have a yubikey set up with some RSA/ECDSA keys. Then find out the public part of you key:

% ssh-keygen -D /usr/local/lib/opensc-pkcs11.so
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCHn4jSqvNLn5NtUxqlAlm1Qj1tlunb0PjBsItmmesquULAM5oqVYmwJ+bmXpDlzgeeIbl1lf3aUTsXCs6My8mkIUwwN+3a5AJ8MA18Bzzx/qIpxe2N/nZ61e03ua5B6SjpfxAAC6i3DPHU6kUSy26sXhfx14y6abWlrwhXaILsTogz2sOganS44au+nSpa35xwMxG5vehkRkYe6vQvvIDFeMhy61DAJGOcGeCbXTfJB9yYwAgOEqTvHDBfbTrHhmnUu82/JV6twY4/tdgrjFxGE3/JsRnoP3lBCoLQR+Uxs3mV7pFelQj/8bZKVsjdzqH9AYWpvIQTJGuwAmyOk61V

This public key should be handed over to your systems administratior (probably yourself in this case) for signing. In return you will get a OpenSSH certificate file that looks sort of a public key but a bit longer.

Now the “tricky” part comes. When you have your public key in ~/.ssh/id_rsa.pub ssh will automatically look for a certificate file called id_rsa-cert.pub. But since we are going to use the smartcard/yubikey to handle our key it will not be visible in ~/.ssh.

First give your certificate a reasonable name like ~/.ssh/yubikey-cert.pub. Then we could tell ssh to combine the two.

Me most basic way is then to just specify the options you need on the command line:

% ssh -o PKCS11Provider=/usr/local/lib/opensc-pkcs11.so -o CertificateFile=~/.ssh/yubikey-cert.pub peter@torus
Enter PIN for 'PIV_II (PIV Card Holder pin)': 

If you want to use this permanently you can of course put the options in ~/.ssh/config instead it should look something like this:

PKCS11Provider=/usr/local/lib/opensc-pkcs11.so
CertificateFile=~/.ssh/yubikey-cert.pub

A third option if you are using the ssh-agent (like me) you could first add the card to your agent:

% ssh-add -s /usr/local/lib/pkcs11/opensc-pkcs11.so
Enter passphrase for PKCS#11: 
Card added: /usr/local/lib/pkcs11/opensc-pkcs11.so

and the specify the cert-file either on the command line or in ~/.ssh/config

% ssh -o CertificateFile=~/.ssh/yubikey-cert.pub peter@torus
[torus:~] peter>

Now you should be able to combine ssh certificates and yubikeys/smart cards

OpenSSH with ECDSA keys on yubikey/smart card

In one of the latest versions of OpenSSH it is now possible to store ECDSA keys on smart cards. If you managed to use ssh with your RSA keys on the yubikey its fairly simple to go to ECDSA. The only thing you need is a late snapshot of OpenSSH portable and a patch.

# fetch http://www.mindrot.org/openssh_snap/openssh-SNAP-20151124.tar.gz
# tar zxf openssh-SNAP-20151124.tar.gz
# cd openssh 
# fetch "https://bugzilla.mindrot.org/attachment.cgi?id=2728" -o ecdsa.patch
# patch -s < ecdsa.patch
# ./configure
[...]
# gmake
[...]

Now you should be able to use get public keys from the smart card on your yubikey. Make sure pcscd is running then fetch the key with your newly compiled ssh-keygen.

# ./ssh-keygen -D /usr/local/lib/opensc-pkcs11.so
C_GetAttributeValue failed: 18
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAJGTLvC9BHSNPAfOw3s4lEix3zKLBKRgZlQ9kSxyttSG8XZ/NIoxm+ZYGbkYxji1kN7brMff21mgXGUzfxp58M=

References:
https://bugzilla.mindrot.org/show_bug.cgi?id=2474
http://lists.mindrot.org/pipermail/openssh-bugs/2015-September/015514.html
https://framkant.org/2015/11/using-smart-card-enabled-yubikey-for-ssh-authentication-in-freebsd/

Using Smart Card enabled yubikey for ssh authentication

This guide will try to show you how to use yubikey for ssh authentication

I recently found out that the yubikey neo had a built in smart card. Also, a few days ago Yubico released their fourth version of the yubikey introducing built in smart card as a standard feature. Me, having very little knowledge and experience with smart cards bought one just to play around with and I quickly found out that the card could be used to securely store encryption keys and certificates.

I use SSH alot both at home and at work, so this was the natural place to start. There is a short guide on how to use the yubikey with ssh on the yubikey developer site but this only covers the yubikey-part. This article will show you how to get things working in FreeBSD. But it will also skip the steps needed to configure the yubikey to have the smart card enabled, and how to generate keys and so on (yubikey software needed)

First if all you need to install a few packages.

# pkg install pcsc-lite
# pkg install opensc
# pkg install libccid

Then there is alot of stuff you can fiddle with using the pkcs11-tool, pkcs15-tool commands. But to get going you can just start the pcscd daemon manually and insert the yubikey into your computer.

# pcscd -f -d

If you have correctly generated a key (and certificate) on the yubikey in one of the slots you should now be able to run

$ ssh-keygen -D /usr/local/lib/opensc-pkcs11.so
C_GetAttributeValue failed: 18
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCOvhEqFq9Ualp1iYiNs0JFs3MgvGU+By/VvyW4qkymW+w/MmHAaHl+/UFnE+kgXChdYHaGEVxxGi6dQlSq+1ZKAWPJsOEbkysK6cgjgvP21gVNjL62TlQz+QfGF82mv0hfSGXQrZQR7VDs+6xJOe3S/0i1HvnnRTdR2v9QSJzd2EWNLmUcPy7+4x4rEB11G0oPt+Xyx60WaleJctwJHHhJS/jqHdvuf7HO6MS/EQn2NTnwIjChlmm2kUT7obnev/r6uEwz87NubnYJUrYImRDMafjS9taq8l7y33ydT00QHEI76kmrSSi7hTfmxUgStQWuQ2mq10YEVd8kZ2sqmC3N

To show the public part of your keys residing on the smart card. If this command was successful you can also add the keys to your ssh-agent using

$ ssh-add -s /usr/local/lib/opensc-pkcs11.so
Enter passphrase for PKCS#11: 
Card added: /usr/local/lib/opensc-pkcs11.so

and then you can list the keys with

$ssh-add -l
2048 8c:c9:99:1f:1b:98:42:af:9b:9b:93:e6:5b:ff:2a:94 /usr/local/lib/opensc-pkcs11.so (RSA)

To remove the card from the agent use

$ ssh-add -e /usr/local/lib/opensc-pkcs11.so
Card removed: /usr/local/lib/opensc-pkcs11.so

To connect somewhere using the keys on the smart card without the agent you can just use

$ ssh -I /usr/local/lib/opensc-pkcs11.so user@host.tld

There will probably be more guides in the future on how to use this further.

Update:

You can now use ECDSA keys stored on yubikey with OpenSSH.