Minimal scp only chroot in FreeBSD

Background

The other day I needed to receive a few files from a friend. And I wanted to provide an easy to use service to do so. Back in the days I did run a FTP service with anonymous login and write acces to a inbox folder (1777).  But these days I have a few more security concerns and wanted to provide a scp drop site, and exactly that. A “scp only chroot”.

Configuration

Giving chrooted ssh access is pretty straight forward in the sshd_config. But to actually get it to work you will have to fiddle around a bit. You will need to figure out exactly what files you will need inside the chroot for the service to function properly.

Create user

Just use your favourite way of creating users in your system.

sshd

The easy part is to configure your sshd_config to chroot the specific user, group or whatever you want to chroot. Since this is probably a one off for me I settled for a single user.

Match User scponly
  ChrootDirectory /home/%u
  X11Forwarding no
  AllowTcpForwarding no

This short configuration will chroot the user scponly to /home/scponly when it connects. You just need to restart sshd in order to these settings to take effect.

Setting up environment

The hardest part was to set up the environment since you will need to make all libs and so on available inside the chroot.

Since this was to be a one off solution I will not go through the hassle of loop/null mounts or stuff like that. I will simply copy the stuff thats actually needed into the chroot. To find out what libs you actually need you can use the tool ldd(1)

# ldd /usr/local/bin/scp 
/usr/local/bin/scp:
	libcrypto.so.8 => /lib/libcrypto.so.8 (0x800a00000)
	libz.so.6 => /lib/libz.so.6 (0x800e69000)
	libutil.so.9 => /lib/libutil.so.9 (0x801082000)
	libldns.so.2 => /usr/local/lib/libldns.so.2 (0x801296000)
	libcrypt.so.5 => /lib/libcrypt.so.5 (0x8014f3000)
	libc.so.7 => /lib/libc.so.7 (0x801712000)

So I just created a few directories in /home/scponly

# cd /home/scponly
# mkdir etc bin lib dev libexec

and then copied these libs to /home/scponly/lib/.

You will also need to have a shell that then can invoke scp. I copied /rescue/sh to /home/scponly/bin/ because it is statically linked and does not depend on any libraries. When all this was done I thought the whole thing was finished but I encountered a few problems.

Problems

The first one was scp complaining about missing ld-elf.so.1. So just like the other libs I just copied it in to /home/sftponly/libexec/.

Secondly scp started to complain about missing /dev/null. And my solution to this was to just create a empty file called null in /home/sftponly/dev/ and chmod to 666. Why it works with a regular file I dont know. If you have any idea please tell me in the comments.

The final problem I faced was an error when invoking scp that said “unknown user 1005”. I did some searches on the web and found several solutions that where Linux specific so no help there. But they involved putting a few more files and libs relevant to nss into the chroot.

What I finally found out was that its the /etc/master.passwd (think of shadow in linux) and its database that where missing. So I just did grep scponly /etc/master.passwd > /home/scponly/etc/master.passwd to copy only the record for scponly to the chroot.

sad:*:1005:1002::0:0:scp only user:/home/scponly:/bin/sh

Finally the actual databased needed to be created with this command

# pwd_mkdb -d /home/scponly/etc/ /home/scponly/etc/master.passwd

Thats it! Now I have a account thats basically all it can do is scp. Yes you can login and get a shell but since there is no other binaries than scp and sh you are pretty limited.

Scalable access control using OpenSSH Certificates

Background

I’ve been using OpenSSH certificates for some time now. They are very handy if you have a bunch of machines you want to trust, or a bunch of machines that shoud trust you.  It’s very effective to trust just one host CA in order to trust all servers with certificates signed by this CA. Or the other way around, have your personal public key signed by a user CA and then be automatically trusted by all servers that trust this CA. But if you working together with alot of people let say within an organisation this becomes problematic pretty soon. Maybe your frontend people should only have access to webservers and database people to the database servers and so on. The solution to this is the little known flags AuthorizedPrincipalsFile and AuthorizedPrincipalsCommand in sshd_config.

OpenSSH Certificates and principals

Lets start with the certificates and the principals within these certificates. In order to make this work I would suggest to use principals within the certificates that are closely tied to the person using it, their company wide username for example. If different keys have different access leves (lets say because on of them are stored on a physical secure element/smart card) it is good to include this kind of information in the principal, according to some standard you make up. Let say I use yubikeys to store my private keys I could have principals like peter_file and peter_physcial. These are easily parsable and  connected to a physical person.

Please note that there could be other access schemes where role is more important that how the key is stored, then role could be a better option for the principal suffix. But please note that if you burn the role into the certificate that person will need to have the current certificate revoked and have a new one issued if the role is ever changed. I will discuss a better way to handle this later.

Another option is to have multiple principals in the same certificate (ie peter,webmaster,root) but this also gets cumbersome when privileges and roles start to change over time.

AuthorizedPrincipalsFile

One solution to this problem is the configuration option AuthorizedPrincipalsFile in sshd_config. With this option you tell sshd where to look for a list of principals valid for a certain user. I looks something like

AuthorizedPrincipalsFile /etc/ssh/%u_principals

When someone tries to log in as peter sshd will check my certificate for validity and then look for valid principals in the file /etc/ssh/peter_principals. sshd expects this file to contain one valid principal per line and optionally preceded by extra options using the same format as the authorized_keys file. (ie from= and command=). This is flexible enough. I can now give multiple principals (or physical persons) access to a specific account by changing a file. I can also restrict access to certain hosts or create force commands for specific principals.

One use case for this could be a webserver where multiple principals(persons) should be able to use the “webmaster” account but at the same time the test/build system should only be allowed to run a certain commands to publish successful builds. Lets say that the account name is www, then the /etc/ssh/www_principals could look something like this:

peter erik from="buildserver.corp.com" command="/bin/publish_website"
buildserver from="guestcomputer.corp.com" guestworker

Please note: If you are really concerned with security, maybe you have given a certificate to a external partner or something, I would suggest to burn the from and command attributes into the certificate. In this way they will never be overridden by some configuration at the server side. The downside is that you will need to produce a new certificate if something changes.

If you have a pretty static setup and/or a decent configuration manager/orchestration tool this could be enough. It gives full flexibility on who should be able to access what, and how. But in the long run it could be tedious to manage all the principal files. This is where the AuthorizedPrincipalsCommand comes in to the picture.

AuthorizedPrincipalsCommand

This works exactly the same as AuthorizedPrincipalsFile but instead of a static file sshd will run a command followed by some options (basically the username that tries to log in) that will generate the principals file dynamically. This gives you a lot of options. Probably the most straight forward one is that you now can have a single ACL file for you whole environment and just let the command read it and produce a host and user specific principal file. One very simple example could look like this:

# Principal(person) user     host
erik                www      webserver.corp.com
peter               www      webserver.corp.com
guestworker         www      webserver.corp.com

peter               db       database.corp.com
erik                db       database.corp.com

Of course this file could be expanded to include more information and more options but this gives a example on how it can be done.

Other backends

But I think the real power in the AuthorizedPrincipalsCommand is that you now can use whatever backend you like and just have the principals command be a wrapper for this backend which could be some Active Directory, LDAP or whatever you might have at your organisation. This makes it possible to use existing infrastructure and still be able to use ssh certificates which I think is a real killer feature in OpenSSH.

If you have some ideas to improve this concept or an questions, please leave a comment.

Setting up a SSH Certificate Authority (CA)

Are you managing a couple of machines over ssh and have begun to feel frustrated about the key management? Find it tedious to distribute your public key to every machine you want to administer? Well, there is a simple solution, that does not include LDAP or some other central authentication server. A little known fact is that OpenSSH have support for both server and client certificates (not x509) since version 5.4. These can be used to set up a trusted Certificate authority on every server once and for all.

This is a very useful tool in environments where server access are harmonized (Where the same set of users should have access to all servers).

The other part is the signing of host keys. This solves the problem of having to manually check and verify the host key fingerprint every time you connect to a new server. If the server key is signed by a CA that you choose to trust you will not be asked to verify the fingerprint. If you have some configuration orchestration like Puppet you can have the puppetmaster create the host certificates on the fly.

From the release notes of OpenSSH 5.4:

* Add support for certificate authentication of users and hosts using a
   new, minimal OpenSSH certificate format (not X.509). Certificates
   contain a public key, identity information and some validity
   constraints and are signed with a standard SSH public key using
   ssh-keygen(1). CA keys may be marked as trusted in authorized_keys
   or via a TrustedUserCAKeys option in sshd_config(5) (for user
   authentication), or in known_hosts (for host authentication).

Creating CA keys
First of all I strongly recommend that you create separate CA keys for hosts and users for security reasons.

The CA keys are just regular ssh keys, you can create them like this:

$ ssh-keygen -a 256 -o -t rsa -b 4096 -f user_ca
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in user_ca.
Your public key has been saved in user_ca.pub.

$ ssh-keygen -a 256 -o -t rsa -b 4096 -f host_ca
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in host_ca.
Your public key has been saved in host_ca.pub.

Securing the private key
Since we are in the process of essentially giving one ssh key access to all of our severs we want keep the private key used for certificate signing very secure. The absolute minimum is to have the key encrypted (password protected). But other than that you should also enable KDF which you can read more about here. I also recommend that you store your keys offline and only have them mounted to a system when you use them. One way could be a encrypted partition on a USB drive, but have two of them since USB drives have a tendency to fail.

Another way to secure your keys is to have them stored on a hardware token, you can read about that here.

Singning your ssh key
First of all you need a regular ssh key which you create with ssh-keygen. Then we can sign this key and create a certificate. This is also done with ssh-keygen:

$ ssh-keygen -s /path/to/ca_key -n peter -z 1234 -V +52w1d-I key_id /path/to/user_key.pub
Signed user key user_key-cert.pub: id "peter_cert" serial 1234 for peter valid from 2016-10-09T22:23:00 to 2017-10-09T22:24:57

The file user_key-cert.pub can now be used together with your private key to access machines that accept the user_ca.pub.

To look at the newly created certificate run:

% ssh-keygen -L -f user_key-cert.pub 
wack-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:AtPyAu1DL5cFruTo9XnsVz7tdec7xF9SbpX8DzsQrbs
        Signing CA: RSA SHA256:8PYQAJojSknTl3BqgBkFKigmaufDhL/7d8zYUNFm7Po
        Key ID: "peter_cert"
        Serial: 1234
        Valid: from 2016-10-09T22:23:00 to 2017-10-09T22:24:57
        Principals: 
                peter
        Critical Options: (none)
        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

Singning host keys
The signing of host keys is done exactly as above, the only difference is that you add -h and the value given to -n should be the hostname.

Configure sshd
To tell sshd to accept key signed with your newly created ca you just need to upload the user_ca.pub and add one line to sshd_config. To tell the server to provide a host certificate to the client you need to add one line per key type to the configuration.

TrustedUserCAKeys /etc/ssh/user_ca.pub
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

and restart sshd.

Configure the client to accept host certificates
To accept host certificates signed by the host_ca you need to add a line to your known_hosts file.
Its just @cert-authority * < public key >

It should look something like this:

@cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3pcm4IJGw76YKZSrf7pZA6s3Hu9eRzKbyzKPgg5OkBWU9ztUz2e1bXtR0UDqQKuOUx+ZDx6wmR3rVRM/BhYt1oeAv6rhxNRW2XPiakpn3EzuEWlbp68QRY8p+k6gjy7cnvF2uyaP0R0Ov819tTMHkdS3Rn57m7pfFk+tuRKLjJnQN2g6uxT+PBJonTQr2XcS5oAzwjel1x7xLqmz0MMrE98uE0GaZHxf/hioXOHt1ihzTF+GPqZ31ZrR3GIWtFEKtvcroaiS25VIV9W39LfVh7RRZTh8oCrjcEdoeFKUKA1PpqblsJhqg1XU2/xa9CAKb6SHb7gwQ18nUaQ/Sk6qYJMgkwWUtXbwS1RIm1k5QlKk8VD4H10jyqiAUBC8SHfNxnqOwKe275AfOVc/iuh4F2NsrIuHfh8tno5LqWSP63D3gXpXXm/4j3Sop2NqNz6EIqoqj0HPIL21/SneJgWXS7xxV/ShgnLVGB0dsJf8PsCnsVt0s5thO22VvE/IVQ1KM/ac1lEafBeSZsuPiOfcIeZu9mm4DfuoyqaHmnV6yBguW6zb894IdcYXsrVnMy3Hp85gnymEZn/qfPJ+dhNbBAANgjWphz5ZaBKdrpgCESz8Ka9S6V7fXr2ikB21YiUKB7XuoPjDncOokSEHU0p5iMQjE+Le7K3nOTtokhrZjXw==

Revoking Certificates
This will be added later.

Protect your private SSH-key with KDF (key derivation function)

Ever heard someone saying that using ssh-keys is a perfect way to have “passwordless” logins to servers? Probably you have. There is a big problem with this approach (I will ignore ssh-agents and stuff like this in this article). If you really want a completley passwordless login, you will need to store the private key unencrypted. The key can be stolen without you knowing it and the attacker could use it “as is”.

The first countermeasure is to encrypt the key and protect it with a passphrase. But since the key is just a file without any brute force protection it could (if the passphrase is weak or semiweak) be very easy to crack the key open. The encryption key used is just a md5 hash of your passphrase and md5 is… fast. Since OpenSSH 6.5 there is a bettery way to protect your ssh-keys. A new private key format is used where you can apply KDF (key deviation function) to slow down the decryption of your private key.

To create a key in the new format with KDF applied you use -o for the new key format and -a specify how many rounds of KDF to use. (more rounds is slower to decrypt)

> ssh-keygen -a 256 -o -t rsa -b 4096 -f test
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in test.
Your public key has been saved in test.pub.
The key fingerprint is:
45:20:9e:50:d2:6e:c9:11:bb:3a:fe:1c:a3:c6:93:48 peter
The key's randomart image is:
+--[ RSA 4096]----+
|    oo+....      |
|     +o+ .       |
|     o+o  .      |
|      =. .       |
|     .. S        |
|  E  .           |
| . oo.o          |
|  ..=+ o         |
|   .ooo          |
+-----------------+

How many rounds to use depends on your environment and how concerned you are about losing your private keys. 256 rounds on a reasonable modern computer takes me ~4s to decrypt. This is a infinite amount of time compared md5 brute force.