Out Of Band Management server

Background
Sometimes I do stupid stuff like editing my firewall rules at home from a remote location and get myself locked out. Sometimes my internet connection is just broken for one reason or the other, this is when you need a out of band channel to your network. You can buy pretty expensive integrated hardware for this with 3G connection and serial consoles and so on, but since this is a project for my home network i decided to build something using a raspberry pi.

Prerequisites
To get this project going I wanted to have a raspberry pi, some sort of wireless connectivity and a serial console to my router.

So I got a Raspberry Pi 2B, this pretty neat case, a old Huawei E1752 from ebay and finally a Linocell Powerbank as a battery backup. For the actual mobile data I got a pre paid SIM card from Telia and got a few GB of data.

Raspberry Pi in case

Raspberry Pi in case

Physical setup and operating system
This setup is very basic and I just put the Pi inside the case and installed FreeBSD using the official image from freebsd.org. I draw power from the powerbank to the pi, and the powerbank is permanently hooked up to power, this way it will run for maybe an hour or so in the event of a power failure.

3G configuration
The reason i got the pretty old E1752 was because it was dirt cheap and also I was absolutley positive it was supported by the u3g driver in FreeBSD.

It is very easy to set up actually, you just put in into a USB port of your Pi and it shows up as three serial interfaces (and maybe some storage device). The first thing you should do is to put the modem in “modem only” mode by sending some AT-codes

# cu -l /dev/cuaU1.0
AT^U2DIAG=0
OK

Then its time to get nostalgic! edit the /etc/ppp/ppp.conf. This was the first time for me since 1998 or something. Of course you will need to figure out some stuff about your 3G provider and make changes accordingly

default:
 set log Phase Chat LCP IPCP CCP tun command
 set device /dev/cuaU1.0
 set timeout 180

telia:
 set speed 115200
 set timeout 0
# set authname wapuser1
# set authkey wap
 set dial "ABORT BUSY TIMEOUT 3 \
        \"\" \
        AT OK-AT-OK \
        AT+CFUN=1 OK-AT-OK \
        AT+CMEE=2 OK-AT-OK \
        AT+CSQ OK \
        AT+CGDCONT=1,\\\"IP\\\",\\\"online.telia.se\\\" OK \
        ATD*99# CONNECT"
 enable dns
 resolv writable
 set ifaddr 10.0.0.1/0 10.0.0.2/0 255.255.255.255 0.0.0.0
 delete! default
 add! default HISADDR

then just test the connection by running “service ppp onestart”

Serial console
My router is located in a small patch cupboard and there was little room for another machine there, so I had to put my OOBM-server somewhere else in my apartment. Luckily I have RJ45 jacks everywhere that are patched to that cupboard so I could very easily run the serial console over the existing CAT6 cables. On the router side I just use a reglar serial cable with DB9 female on one side and a RJ45 male on the other side. On the OOBM-server side I have a simple USB-serial converter followed by a DB9 female to RJ45 female converter. The USB-serial converter shows up in FreeBSD as a regular serial interface like /dev/cuaU0.

RJ45 to DB9

RJ45 to DB9

Out Of Band functionality
Lets put everything together. The first thing i needed to figure out was how to enable the 3g connection remote, but this was pretty simple because the modem can receive sms messages. So I just send some magic/secret sms to the modem that tells it to connect.

Next problem I encountered was that Telia blocks all (?) incoming ports on the mobile connection and since I want to do ssh based administration this was a problem. To work around this problem I went for a solution where the OOBM-server first sets up the PPP connection and then sets up a ssh connection with remote port forwarding to one of my amazon instances. Then I just ssh to the amazon instance on some port and end up on “localhost” on the OOBM-server. On the amazon instance I have created a specific account only used for this purpose that accepts the ssh key used by the OOBM-server.

To put everything together I wrote a small python script that runs every few minutes and checks for valid sms messages on the modem and if it finds such message fires up the PPP connection and then the ssh connection. I will spare you the hazzle of reading my ugly code but here is some pseudo code describing what it does:

m = connect(modem)
if m.send_at("AT") != "OK":
   print "modem is not responsive"
   exit(1)

#Look for valid activation sms
for msg in m.messages():
    if msg.number == "NYNUMBER" && msg.text == "SECRET"
        activate = True
        break

#Try a maximum of three times to set up the connection.
if activate:
    for tries in range(3):
        start_ppp()

        #Check that we can reach internet
        test_connectivity()

        #check that we actually  reach internet via 3G
        verify_route() 
     
        # set up the reverse ssh (ssh -R 31337:localhost:22 remote_server)
        start_ssh()
     
        #Notify me via pushover that connection is up
        send_push("OOBM link up")
 
        sleep(1800)

        m.delete_message(all)
        stop_ssh()
        stop_ppp()
    
        #Notify me via sms that OOBM link is down
        m.send_sms(NUMBER,"OOBM link is down")
        exit(0)

I have omitted the error handling in the pseudo code but I ensure you that the actual script have some.. 😉

The reason i use push messages when the link is up and sms when the link is down is because this modem doesnt have multiplexing and it cant send sms messages while connected.

When the connection is up and running its a simple task to just ssh to remote_server at port 31337 and then login to the OOBM server. From there you can do further ssh connection from the inside of your network or just use the serial console to talk to the router.

% ssh root@remote_server -p31337
root@oobm:~ #
root@oobm:~ # cu -s 57600 -l /dev/cuaU0
Connected

FreeBSD/i386 (gw) (ttyu0)

login:

This is how the setup looks like:

Network diagram

Network diagram

Here is the server installed at its current location

OOBM server on top of power bank

OOBM server on top of power bank

Stratum 1 NTP server with FreeBSD on Raspberry Pi.

Background
We where frustrated at work about the central IT organisation blocking outgoing 123/udp connections. This gave us the idea to buy a GPS controlled NTP server like this one: https://www.meinbergglobal.com/english/products/rack-mount-1u-ntp-server.htm. While doing some research on this subject I found that alot of people seem to build their own stratum 1 NTP servers at home. So I decided to build one myself to use at home.

Prerequisites
There is alot of information floating around on this subject but I wanted to use FreeBSD and Raspberry Pi. It took me a while to figure this out but I bought this set of hardware:
Raspberry Pi 2 Model B
Adafruit Raspberry Pi Case
Adafruit Ultimate GPS Breakout
Adafruit Perma-Proto HAT
External antenna

Basic setup
There is now a new kernel module named gpiopps written by ian@freebsd.org that you can use to get PPS input on any gpio pin. To configure what gpio pin to use you need to rebulid the device tree of your raspberry pi (/usr/src/sys/boot/fdt/dts/arm/rpi2.dts). This is the changes i made:

# svnlite diff
Index: rpi2.dts
===================================================================
--- rpi2.dts	(revision 309114)
+++ rpi2.dts	(working copy)
@@ -337,6 +337,13 @@
 		broadcom,depth = ;		/* Set by VideoCore */
 	};
 
+
+ 	pps@0  {
+		compatible = "pps-gpio";
+		gpios = <&gpio 17 0>;
+		status = "okay";
+	};
+
 	leds {
 		compatible = "gpio-leds";

and then rebuilt the tree

# cd /usr/src/sys/tools/fdt
# setenv MACHINE arm
# ./make_dtb.sh /usr/src/sys /usr/src/sys/boot/fdt/dts/arm/rpi2.dts rpi2.dtb
# cp rpi2.dtb /boot/msdos

Then you can hook up the pps output of the GPS to gpio pin 17 of your raspberry pi and make sure gpiopps is loaded by adding this line to /boot/loader.conf.

gpiopps_load="YES"

Reboot the pi and then you should see something like this in the boot messages

gpiopps0:  on ofwbus0
gpiopps0: PPS input on gpio0 pin 17

Physical setup
For the physical setup I used the Pi, a case and the “perma proto hat” and did some simple soldering to hook up the serial interface of the GPS to the uart serial interface of the Pi, and of course the PPS output to gpio pin 17. I also added a LED to the pps output so I can visually see when I have a working PPS signal.

Soldered proto hard

Soldered proto hard

GPS soldered to the proto hat

GPS soldered to the proto hat

Finished product:

NTP server

GPS configuration
The GPS is actually very easy to talk to. It has a serial interface configured to 9600 baud by default and a dedicated PPS output.

To control the GPS you can send basic text-strings to the unit, for example setting the update rate to 1Hz:

# printf '$PMTK251,57600*2C\r\n' > /dev/cuau0 (Set baudrate to 57600)
# printf '$PMTK314,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0*29\r\n' > /dev/cuau0 (ask only for GPRMC sentences)
# printf '$PMTK220,1000*1F\r\n' > /dev/cuau0 (echo sentences once a second)
# printf '$PMTK300,1000,0,0,0,0*1C\r\n' > /dev/cuau0 (update fix once a second)

If you want a complete set of instructions(commands) you can send to device please see the command set sheet.

NTP configuration
I have four different kind of time sources configured on my stratum 1 server.

1. External time sources (internet). In Sweden we have a pretty neat project at ntp.se where they have built custom ntp-servers using FPGAs and atomic clocks.They should be able to server time at 10Gbit/s line rate. You can read more about this here: http://www.ntp.se. I use all of these servers as reference clocks.

2. Local servers that run ntpd. Just my gateway and my server. In the case that all other references fail they can still discipline each other.

3. PPS input from the GPS module. This is the main thing about this article. The GPS outputs a pulse every second that is then used to discipline ntpd.

4. GMEA data from the GPS module. The GPS also outputs coordinates and times on the serial console. But these timestamps are pretty imprecise, at best its close within a second. (But since we also have PPS this is good enough)

You can see my ntp.conf here below but I will only talk about the GPS-stuff from now on.

I use two different drivers in ntpd. 20 and 22. These are the NMEA and PPS drivers. My settings for the PPS driver is pretty basic, it will automatically look for /dev/pps0 and try to fix to a pps signal. The gpiopps driver creates gpiopps0 so I have added “link gpiopps0 pps0” to /etc/devfs.conf

For the NMEA driver we have a few more settings. First of all “mode 17” sets what type of output the look for from the GPS and what baudrate to use. From the driver documentation you find that bit 0 is used to set processing of $GPMRC sentences from the GPS. Bit 4-6 is used to set baudrate and decimal “16” is 9600. So 9600+$GPMRC=17 right? 🙂 This driver looks for /dev/gps0 by default so I have added “link cuau0 gps0” to /etc/devfs.conf

The fudge time2 is used to compensate for the delays we have in the serial interface of the GPS to make it match PPS more closely.

Please refer to the driver documentation for more settings:
http://doc.ntp.org/4.2.8p8/drivers/driver20.html
http://doc.ntp.org/4.2.8p8/drivers/driver22.html

# Allow traffic to external servers
restrict 194.58.203.20 mask 255.255.255.255 nomodify notrap noquery
restrict 194.58.203.148 mask 255.255.255.255 nomodify notrap noquery
restrict 194.58.204.20 mask 255.255.255.255 nomodify notrap noquery
restrict 194.58.204.148 mask 255.255.255.255 nomodify notrap noquery
restrict 194.58.202.20 mask 255.255.255.255 nomodify notrap noquery
restrict 194.58.202.148 mask 255.255.255.255 nomodify notrap noquery
restrict 194.58.205.20 mask 255.255.255.255 nomodify notrap noquery
restrict 194.58.205.148 mask 255.255.255.255 nomodify notrap noquery

# Allow traffic to internal servers
restrict 172.25.0.25 mask 255.255.255.255 nomodify notrap noquery
restrict 172.25.0.1 mask 255.255.255.255 nomodify notrap noquery


# PPS
server 127.127.22.0 flag3 0

# NMEA
server 127.127.20.0 mode 17
fudge 127.127.20.0 time2 +0.767

# Servers
server 194.58.203.20 iburst  prefer
server 194.58.203.148 iburst
server 194.58.204.20 iburst
server 194.58.204.148 iburst
server 194.58.202.20 iburst
server 194.58.202.148 iburst
server 194.58.205.20 iburst
server 194.58.205.148 iburst
server 172.25.0.25
server 172.25.0.1

driftfile /var/db/ntp.drift

# save ntp performance stats
statistics loopstats
statsdir /var/log/ntp/
filegen peerstats file peerstats type day enable
filegen loopstats file loopstats type day enable
filegen clockstats file clockstats type day enable

Running ntpd
So now when we have configured ntpd how does it actually work? This is some output from ntpq -p efter some 30mins of ntpd running:

root@ntp:/dev # ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 torus.pean.org  172.25.0.10      2 u   20   64  377    0.417    0.816   0.033
 gw.pean.org     172.25.0.10      2 u   18   64  377    0.609    0.462   0.102
oPPS(0)          .PPS.            0 l    5   64  377    0.000    0.133   0.011
xGPS_NMEA(0)     .GPS.            0 l    4   64  377    0.000   62.005   6.586
*gbg1.ntp.se     .PPS.            1 u   21   64  377    7.467    0.052   0.054
+gbg2.ntp.se     .PPS.            1 u   25   64  377    7.514    0.167   0.074
+mmo1.ntp.se     .PPS.            1 u   14   64  377   11.469    0.127   0.221
+mmo2.ntp.se     .PPS.            1 u   22   64  377   11.386    0.152   0.139
+sth1.ntp.se     .PPS.            1 u   12   64  377    2.087    0.130   0.235
+sth2.ntp.se     .PPS.            1 u   17   64  377    2.255    0.158   0.292
-svl1.ntp.se     .PPS.            1 u   13   64  377    6.686   -0.118   1.343
+svl2.ntp.se     .PPS.            1 u    6   64  377    6.015    0.144   1.177

The o in oPPS denotes that ntpd have PPS signal. The x in xGPS_NMEA denotes that this source is marked as a false ticker. The reason for this could be that I have entered a to large fudge factor, its seem to be running 62ms fast at this point. I will keep ntpd running for a few hours and the try to adjust the time fudge accordingly.

Get rid of your fiber converter (tranciever)

Background
I live in a so called “Telia house” (internet, tele and tv are supplied by telia). These services are provided via fiber at my house, and its actually fiber all the way to the patch panel in my apartment. The connection is then coverted from 1000Base-X to 1000Base-T using a media converter from CTS (HET-3012). Maybe a year ago the media converter failed and my connection was down for almost two weeks. It was at that time I got the idea to get rid of the media converter and have the incoming fiber directly connected to my router.  This way I could get rid of one possible cause of failure and also save som space in my already packed patch cupboard.

Prerequisites
First of all I needed a network card that could fit into my router (a Soekris 6501-50). The soekris have a regular PCI Express 1x interface so it was pretty easy to get a card from ebay. I got a LR-Link LREC6230PF-SFP which used the Intel I210 chip which is well supported in FreeBSD by the igb driver.

Then I needed to find the correct SFP for my application. My first problem was that my incoming fiber uses a single fiber and not a pair, I had never seen something like this. But after some reading on the internet and a quick look at the CTS documentation i found that it uses 1310nm for TX and 1550nm for RX and SC contact. After that is was a simple task to just get a SFP with the correct specs from https://www.direktronik.se

Installation and configuration
When I had the hardware I needed there was no problem installing it. The network card fitted quite nicely in the pretty small case of my Soekris and the SFP worked out of the box.

Soekris
bidi SFP

In order to load the driver for this specific card at boot I added this line to /boot/loader.conf

if_igb_load="YES"

Since I wanted to keep my “old” IP adress I configured the same mac-adress on the new card as on the old card. (And yes, I also changed the mac of the old card to avoid conflicts).
rc.conf:

ifconfig_igb0="DHCP ether 00:00:24:d0:XX:XX description 'Internet uplink'"

And this is how the final result looks like inside the patch cupboard. The three black cables are antenna cables for the wifi. You can read more about the general setup of my router here.
Connected fiber

Use a smart card or HSM to securely store your SSH CA keys

Depending on the use case, SSH CA keys can be extremely sensitive. Possession of the private key gives you the ability to sign new certificate for arbitrary usernames that will grant access to all machines where this CA is trusted. This is why you should keep your CA keys very safe, preferably offline and of course encrypted (password protected).

One big problem with digital encryption keys is that it is extremely difficult to find out if you lost control over they keys. If they where copied you have no way of finding out. This is where hardware tokens comes in. If you can be sure that the key will never leave the hardware token, you have a better chance of knowing when they keys are lost or stolen.

OpenSSH have support for storing keys onto hardware tokens that talk PKCS#11 such as smart cards and hardware security modules.

If you want to store your personal SSH key on a smart card you can read more about that here: Using Smart Card enabled yubikey for ssh authentication in FreeBSD

Since a SSH CA is just a regular SSH key they can also be stored on smart cards. But there is a few subtle differences in how you use it. First of all you need a token with a RSA key present. There is no option in OpenSSH to ask for a specific key stored on the token, instead we specifiy the public key that matches the private key we want to use. Lets download the public key from the token:

$ ssh-keygen -D /usr/local/lib/opensc-pkcs11.so
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCOvhEqFq9Ualp1iYiNs0JFs3MgvGU+By/VvyW4qkymW+w/MmHAaHl+/UFnE+kgXChdYHaGEVxxGi6dQlSq+1ZKAWPJsOEbkysK6cgjgvP21gVNjL62TlQz+QfGF82mv0hfSGXQrZQR7VDs+6xJOe3S/0i1HvnnRTdR2v9QSJzd2EWNLmUcPy7+4x4rEB11G0oPt+Xyx60WaleJctwJHHhJS/jqHdvuf7HO6MS/EQn2NTnwIjChlmm2kUT7obnev/r6uEwz87NubnYJUrYImRDMafjS9taq8l7y33ydT00QHEI76kmrSSi7hTfmxUgStQWuQ2mq10YEVd8kZ2sqmC3N

We put this into a file, let say ssh_ca.pub. Now we are ready to start using the token for certificate signing.

$ ssh-keygen -s ssh_ca.pub -D /usr/local/lib/opensc-pkcs11.so -I key_id wack.pub 
Enter PIN for 'Framkant HSM Test (UserPIN)': 
Signed user key wack-cert.pub: id "key_id" serial 0 valid forever

To look at the certificate we can use ssh-keygen.

% ssh-keygen -L -f wack-cert.pub 
wack-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:AtPyAu1DL5cFruTo9XnsVz7tdec7xF9SbpX8DzsQrbs
        Signing CA: RSA SHA256:Pbdx6TAvXvwZTKQVMRYWsWYPomw6AxBVoqbtXgy9pfs
        Key ID: "key_id"
        Serial: 0
        Valid: forever
        Principals: (none)
        Critical Options: (none)
        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

In a real life scenario I would recommend against this kind of certificate. Give it some validity and specify principals.

Setting up a SSH Certificate Authority (CA)

Are you managing a couple of machines over ssh and have begun to feel frustrated about the key management? Find it tedious to distribute your public key to every machine you want to administer? Well, there is a simple solution, that does not include LDAP or some other central authentication server. A little known fact is that OpenSSH have support for both server and client certificates (not x509) since version 5.4. These can be used to set up a trusted Certificate authority on every server once and for all.

This is a very useful tool in environments where server access are harmonized (Where the same set of users should have access to all servers).

The other part is the signing of host keys. This solves the problem of having to manually check and verify the host key fingerprint every time you connect to a new server. If the server key is signed by a CA that you choose to trust you will not be asked to verify the fingerprint. If you have some configuration orchestration like Puppet you can have the puppetmaster create the host certificates on the fly.

From the release notes of OpenSSH 5.4:

* Add support for certificate authentication of users and hosts using a
   new, minimal OpenSSH certificate format (not X.509). Certificates
   contain a public key, identity information and some validity
   constraints and are signed with a standard SSH public key using
   ssh-keygen(1). CA keys may be marked as trusted in authorized_keys
   or via a TrustedUserCAKeys option in sshd_config(5) (for user
   authentication), or in known_hosts (for host authentication).

Creating CA keys
First of all I strongly recommend that you create separate CA keys for hosts and users for security reasons.

The CA keys are just regular ssh keys, you can create them like this:

$ ssh-keygen -a 256 -o -t rsa -b 4096 -f user_ca
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in user_ca.
Your public key has been saved in user_ca.pub.

$ ssh-keygen -a 256 -o -t rsa -b 4096 -f host_ca
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in host_ca.
Your public key has been saved in host_ca.pub.

Securing the private key
Since we are in the process of essentially giving one ssh key access to all of our severs we want keep the private key used for certificate signing very secure. The absolute minimum is to have the key encrypted (password protected). But other than that you should also enable KDF which you can read more about here. I also recommend that you store your keys offline and only have them mounted to a system when you use them. One way could be a encrypted partition on a USB drive, but have two of them since USB drives have a tendency to fail.

Another way to secure your keys is to have them stored on a hardware token, you can read about that here.

Singning your ssh key
First of all you need a regular ssh key which you create with ssh-keygen. Then we can sign this key and create a certificate. This is also done with ssh-keygen:

$ ssh-keygen -s /path/to/ca_key -n peter -z 1234 -V +52w1d-I key_id /path/to/user_key.pub
Signed user key user_key-cert.pub: id "peter_cert" serial 1234 for peter valid from 2016-10-09T22:23:00 to 2017-10-09T22:24:57

The file user_key-cert.pub can now be used together with your private key to access machines that accept the user_ca.pub.

To look at the newly created certificate run:

% ssh-keygen -L -f user_key-cert.pub 
wack-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:AtPyAu1DL5cFruTo9XnsVz7tdec7xF9SbpX8DzsQrbs
        Signing CA: RSA SHA256:8PYQAJojSknTl3BqgBkFKigmaufDhL/7d8zYUNFm7Po
        Key ID: "peter_cert"
        Serial: 1234
        Valid: from 2016-10-09T22:23:00 to 2017-10-09T22:24:57
        Principals: 
                peter
        Critical Options: (none)
        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

Singning host keys
The signing of host keys is done exactly as above, the only difference is that you add -h and the value given to -n should be the hostname.

Configure sshd
To tell sshd to accept key signed with your newly created ca you just need to upload the user_ca.pub and add one line to sshd_config. To tell the server to provide a host certificate to the client you need to add one line per key type to the configuration.

TrustedUserCAKeys /etc/ssh/user_ca.pub
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

and restart sshd.

Configure the client to accept host certificates
To accept host certificates signed by the host_ca you need to add a line to your known_hosts file.
Its just @cert-authority * < public key >

It should look something like this:

@cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3pcm4IJGw76YKZSrf7pZA6s3Hu9eRzKbyzKPgg5OkBWU9ztUz2e1bXtR0UDqQKuOUx+ZDx6wmR3rVRM/BhYt1oeAv6rhxNRW2XPiakpn3EzuEWlbp68QRY8p+k6gjy7cnvF2uyaP0R0Ov819tTMHkdS3Rn57m7pfFk+tuRKLjJnQN2g6uxT+PBJonTQr2XcS5oAzwjel1x7xLqmz0MMrE98uE0GaZHxf/hioXOHt1ihzTF+GPqZ31ZrR3GIWtFEKtvcroaiS25VIV9W39LfVh7RRZTh8oCrjcEdoeFKUKA1PpqblsJhqg1XU2/xa9CAKb6SHb7gwQ18nUaQ/Sk6qYJMgkwWUtXbwS1RIm1k5QlKk8VD4H10jyqiAUBC8SHfNxnqOwKe275AfOVc/iuh4F2NsrIuHfh8tno5LqWSP63D3gXpXXm/4j3Sop2NqNz6EIqoqj0HPIL21/SneJgWXS7xxV/ShgnLVGB0dsJf8PsCnsVt0s5thO22VvE/IVQ1KM/ac1lEafBeSZsuPiOfcIeZu9mm4DfuoyqaHmnV6yBguW6zb894IdcYXsrVnMy3Hp85gnymEZn/qfPJ+dhNbBAANgjWphz5ZaBKdrpgCESz8Ka9S6V7fXr2ikB21YiUKB7XuoPjDncOokSEHU0p5iMQjE+Le7K3nOTtokhrZjXw==

Revoking Certificates
This will be added later.

Publish certificate fingerprints in DNS (TLSA/DANE)

Ever had second thoughts on paying a certificate authority (CA) a lot of money to sign your web servers public encryption key to get a trusted certificate? With the birth of DNSSEC the need for this could fade away, at least partly. It is now possible to create a self signed certificate and post the signatures securely in DNS. This way there is a secure out of band method to check the validity of a certificate. There is a special record type for this called TLSA and you can read more about it in RFC 6698

The support for these records in todays browsers are very limited, but there is a plugin available for the most popular browsers called DNSSEC validator. With this plugin it is possible to get a green light, even with a self signed certificate.

If you run your own mail server it could be interesting to know that Postfix have support for TLSA records since a few versions back.

A TLSA record could look like this:

_443._tcp TLSA (3 0 2 4FB72400493E364A499B24CDC5E5715F
                      97543262CBCB90C8483C5AB3E8A37C9E
                      CC4E021C8C12B3E485CFF3A082348FE6
                      ED39EBBF2F812B3BA8857DBB1C96AFF0)

_443._tcp tells us that a certificate with this sha-512 hash should be handed to us if we connect to tcp port 443.

There are three options before the hash. The first option defines “certificate usage”, the second “TLSA selector” and the third is basically hash type. The fourth field is the actual hash of the certificate. In the above example we have a sha-512 of the full certificate of a “Domain-issued certificate”. Please read more about this in the RFC (section 7).

Using the *nix command host the fetch this record looks like this

> host -t tlsa _443._tcp.framkant.org.
_443._tcp.framkant.org has TLSA record 3 0 2 4FB72400493E364A499B24CDC5E5715F97543262CBCB90C8483C5AB3 E8A37C9ECC4E021C8C12B3E485CFF3A082348FE6ED39EBBF2F812B3B A8857DBB1C96AFF0

It is pretty easy to find out the hash of a certificate using openssl. The following command gives us the sha512 hash of a certificate from file.

> openssl x509 -noout -fingerprint -sha512 -in framkant.crt | tr -d :

Remember that for this to have any effect on your security or your ability to have self signed certificates you need to have DNSSEC up and running for your domain. Please have a look at my article about OpenDNSSEC if you run your own authoritative dns server.

Protect your private SSH-key with KDF (key derivation function)

Ever heard someone saying that using ssh-keys is a perfect way to have “passwordless” logins to servers? Probably you have. There is a big problem with this approach (I will ignore ssh-agents and stuff like this in this article). If you really want a completley passwordless login, you will need to store the private key unencrypted. The key can be stolen without you knowing it and the attacker could use it “as is”.

The first countermeasure is to encrypt the key and protect it with a passphrase. But since the key is just a file without any brute force protection it could (if the passphrase is weak or semiweak) be very easy to crack the key open. The encryption key used is just a md5 hash of your passphrase and md5 is… fast. Since OpenSSH 6.5 there is a bettery way to protect your ssh-keys. A new private key format is used where you can apply KDF (key deviation function) to slow down the decryption of your private key.

To create a key in the new format with KDF applied you use -o for the new key format and -a specify how many rounds of KDF to use. (more rounds is slower to decrypt)

> ssh-keygen -a 256 -o -t rsa -b 4096 -f test
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in test.
Your public key has been saved in test.pub.
The key fingerprint is:
45:20:9e:50:d2:6e:c9:11:bb:3a:fe:1c:a3:c6:93:48 peter
The key's randomart image is:
+--[ RSA 4096]----+
|    oo+....      |
|     +o+ .       |
|     o+o  .      |
|      =. .       |
|     .. S        |
|  E  .           |
| . oo.o          |
|  ..=+ o         |
|   .ooo          |
+-----------------+

How many rounds to use depends on your environment and how concerned you are about losing your private keys. 256 rounds on a reasonable modern computer takes me ~4s to decrypt. This is a infinite amount of time compared md5 brute force.

Publish ssh public host key fingerprints in DNS (SSHFP)

This article describes a solution to the problem where people answers ‘yes’ on the question if they are sure they want to continue connection to a new/unknown SSH server.

Have you ever blindly answered yes to something like this?

The authenticity of host 'framkant.org (52.16.239.146)' can't be established.
ECDSA key fingerprint is 42:f9:b3:40:17:61:89:e4:80:cf:d5:ae:bb:bb:7d:75.
+--[ECDSA  256]---+
|   ....o+o       |
|  .  oooo.       |
|   o .=..        |
|    oo o.        |
|      o.S        |
|      .o o  . E  |
|       ..  . .   |
|      ..  .      |
|      ++..       |
+-----------------+
Are you sure you want to continue connecting (yes/no)? 

Well, dont. Of course its very important to know WHERE you are connecting. In a directed attack the attacker could have set up a server of their own to fool you to reveal important passwords or other information. But most people dont think to much about this and just answers yes to get rid of the message.

The problem is, HOW do you distribute the correct fingerprint and check it? In a datacenter there are many was to do this (configuration orchestration, ldap and so on) but to end users and home computers this is a probem. Of coruse you could post your fingerprints on a webpage secured with https, but the chances a user would really sit down and compare the numbers are slim.

The solution is DNSSEC. When your zone is signed and the chain of trust is complete a client can really trust the correctness of the information present in the DNS tree. Publishing your ssh host key fingerprints here sounds like a great idea. The new record type SSHFP solves this problem and the good news is that OpenSSH have support since a few years back. Its even so simple that ssh-keygen can generate the new records for you:

> ssh-keygen -r framkant.org
framkant.org IN SSHFP 1 1 d6a64454a9a559ec35f59994c2e5f8376ff86ac8
framkant.org IN SSHFP 1 2 99d78eff62823561bab2661a72250f1e8344c1c6a4e8903f0d5d8ebce6d819b7
framkant.org IN SSHFP 2 1 9572024308e542074768bbfd6a3ff8b30e940ce2
framkant.org IN SSHFP 2 2 ab63e4ebe5ba0b9eb12a9735a533cfdf5e2807bac852e8b019dedfb630841896
framkant.org IN SSHFP 3 1 fe1160646ac00872d269ebd7ddaa07cb83d7d6af
framkant.org IN SSHFP 3 2 6580debbe37b4a2f32eed78aec2e2f943de6559f24f2b6d7671e78a0ca469728

The first digit represent the key type and the second represent the hash used. You can read more about this in RFC 4255. If your zone is signed by DNSSEC and the chain of trust is complete you will see something like this when connecting to a server with SSHFP records attached to it.

debug1: Server host key: ECDSA 42:f9:b3:40:17:61:89:e4:80:cf:d5:ae:bb:bb:7d:75
debug1: found 6 secure fingerprints in DNS
debug1: matching host key fingerprint found in DNS
debug1: ssh_ecdsa_verify: signature correct

OpenSSH uses two different methods to find out if the records are secured by DNSSEC. If compiled with ldns-support it will perform the DNSSEC-validation itself. If not, it will trust the resolvers in resolv.conf to do the validation. The resolver will set a special bit, the AD-bit (Authenticated Data) if dnssec validates. If the AD-bit is set on the response from the resolver ssh will assume that the records can be trusted.

If you want ssh to always try to use information found in DNS add this to your ~/.ssh/config

Host *
    VerifyHostKeyDNS yes

References
RFC 4255

WPA2 Enterprise with hostapd

Secure your wireless network with WPA2 Enterprise.

I’ve had a really hard time finding any good documentation on hostapd, but I finally got it to work. So why would you use WPA2 Enterprise, isnt Personal good enogh? Well, the problem is these days that anyone can set up a wireless network with the same name as yours. If you connect to this fake network beliving you are connected to your own someone have full access to everything you send and receive. WPA2 Enterprise fixes this because the access point also has to prove its identity by providing a valid SSL Certificate. If this certificate changes you will be notified right away.

So, how do you set it up in hostapd? Well, its not that complex at all, the problem is that you cant find any documentation on it.

Here is my configuration for WPA2 Enterprise. This uses hostapds own internal RAIDUS-server but you could as well use FreeRADIUS or some other RADIUS server out there, but then you’ll have to figure out how to configure that in hostapd. 😀

/etc/hostapd.conf

interface=wlan0
driver=bsd
ctrl_interface=/var/run/hostapd
ctrl_interface_group=wheel
ssid=web
wpa=2
wpa_key_mgmt=WPA-EAP
wpa_pairwise=TKIP CCMP
macaddr_acl=0 
auth_algs=1
own_ip_addr=127.0.0.1
ieee8021x=1
eap_server=1
eapol_version=1

# Path for EAP server user database
eap_user_file=/etc/hostapd_eap_user
ca_cert=/etc/ssl/startssl.ca.pem
server_cert=/etc/ssl/domain.tld.crt
private_key=/etc/ssl/domain.tld.key

/etc/hostapd_eap_user

"peter@domain.tld" PEAP [ver=0]
"peter@domain.tld" MSCHAPV2 "passphrase" [2]

You can also set up it to require client certificate to connect to make it even more “secure”. It all depends on how you handle your certificates and passwords.

#"peter@domain.tld" PEAP [ver=0]
"peter@domain.tld" TLS

I use free host and client certificates from https://www.startssl.com but if you want to set up your own CA you can find and excellent article here
Update:
I have found some “documentation” in the form of well commented configurationfiles.
https://w1.fi/cgit/hostap/plain/hostapd/hostapd.conf
https://w1.fi/cgit/hostap/plain/hostapd/hostapd.eap_user
https://w1.fi/cgit/hostap/plain/hostapd/

Secure wireless FreeBSD router

One of my hobby projects is to build a “fully fledged” wireless router running FreeBSD. To do this I got a Soekris 6501-50 which has four gigabit ports and two mini PCIe slots.

For storage I have a small 30GB miniPCIe SSD (OCZ-NOCTI). I have two wireless cards installed one dual band 3x3MIMO minipcie card from COMPEX (When FreeBSD gets up to speed with 802.11ac I will upgrade to a WLE900VX) and for legacy 2.4Ghz I use a TPLINK TL-WN781ND. These two cards are then used for two virtual interfaces each, one for my primary wlan and one for my guest wlan.

Atennas
One important part of the wireless part of the router is the antennas. For a long time I used three consumer antennas (ALLNET ALL19003) for 5GHz use for my COMPEX card, but when I moved to a new apartment and switched to 5GHz I didnt get good speeds everywhere in the apartment. This got me thinking about a ceiling antenna. But I found it very hard to find one in consmer stores, so I turned to Ebay. I found a decently cheap Laird S2451DBT 6port antenna which is now mounted in the ceiling at a central point of the apartment.

Firewall
To do packet filtering I use the OpenBSD derived pf(4) which has been in FreeBSD for a long time now.

Wireless networks
Like I told you in earlier I have two physical cards and two networks (ssid). One primary and one guest networks with tighter rules but less strict authentication.

The firt step here is to find a card supported by FreeBSD, but also have support for hostap mode, obviously mine does. To get it to run WPA2 Personal on a specific channel the configuration is very easy

interface=wlan2
ctrl_interface=/var/run/hostapd
ctrl_interface_group=wheel
ssid=web2
wpa=2
wpa_passphrase=
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP

Lately I started to use the more secure WPA2 Enterprise and you can read about my configuration here

Other networks
My internet provider is Telia and I have something they call Triple Play.. (Internet, TV and Tele in the same cable). In order to get this to work you should use their Thompson router that has som magic for splitting up your one incoming fiber into three different networks. After some reading on the interwebs I found out this is really just tagged vlans and I can split them myself without any problem. “Internet” comes untagged and the IPTV traffic comes in on tag 845, so I just create a vlan interface with my external interface as parent and then bridge this interface to one of my internal interfaces on the Soekris. (and of coruse connect the IPTV-box to this port). You can read more about this configuration here (swedish)

This is how the interfaces are configured
home_networkhome_network

home_network