Category Archives: Sysadmin

How to configure 802.1X client and server in FreeBSD

This article describes the steps required to configure 802.1X client and server using EAP-TLS both client and server side in FreeBSD.

Background

I recently bought a new switch of ebay capable of 802.1X PNAC (Port based Network Access Control). I wanted to have this set up for a long time, but it wasn’t unil now I had a switch thats actually supported it. This article describes how I got it up and running.

Certificates

Since I already have EAP-TLS set up for my wifi (authentication using X.509 client certificates) I will also use EAP-TLS for wired access. So I configured a private CA in order to issue both server certificates for the radius server and also client certificates for all clients that will use my network.  I have a more general post about how set up the CA here.

Radius server (Authentication server)

You will need to configure a Radius server to handle the authentication requests. I already have EAP-TLS configured using hostpad and its internal radius server for my wifi. But that server is very limited, so I decided to give FreeRADIUS a go. This also means that my wifi clients will be authenticated using the FreeRADIUS server from now on.

I understand that FreeRADIUS is very flexible and “easy” to customize, but I really think that the configuration is very hard do grasp. It would be virtually impossible to configure it without some guide to follow. The two big problems are that the configuration is split up into MANY files and that all the documentation is inside the config files, which makes them really hard to read. Luckily I found this guide online that did exactly what I wanted. So please have a look at that guide under “Configuration” to see how I configured FreeRADIUS. Its basically just a few minor changes in four files.

The switch (Authenticator)

This article will not cover the switch configuration needed for this setup. The configuration you will have to do is very depended on what brand of switch you have and what software it is running.  I have a Juniper EX2200-C and there is good online documentation on how to set up 802.1X.

The Supplicant (802.1X client)

In 802.1X the client is called the supplicant. To authenticate against the radius server you will basically need a small supplicant software installed on the client that will handle the authentication. This is done using EAPOL-packages that are sent out on the network and then handled by the switch (The Authenticator). The switch then talks to the raidus server (The Authentication server) to verify the client.

In Linux and FreeBSD the most commonly used supplicant software is called wpa_supplicant. Most of you who know of wpa_supplicant have used it for wifi authentication in differents forms. It can handle alot of different security types like WPA2 Enterprise, WPA2 or even WEP. But it can also work with wired network authentication. The configuration is actually very straight forward and similar to the wifi configs.

network={
        key_mgmt=IEEE8021X
        eap=TLS
        identity="identity"
        ca_cert="/etc/ssl/chain.pem"
        client_cert="/etc/ssl/client.cert.pem"
        private_key="/etc/ssl/client.key"
        private_key_passwd="passw0rd"
}

This is all you will need to have wpa_supplicant authenticate using client certificates over ethernet.

To have the wpa_supplicant automatically started when you FreeBSD machine boots you can just add the WPA keyword to your interface declaration in /etc/rc.conf like this:

ifconfig_ue0="DHCP WPA"

Smart Card/HSM backed OpenSSL CA

This article describes how to set up a Smart Card/HSM backed OpenSSL CA using a Smart Card HSM or any PKCS11 enabled device.

Background

Since some years back I use WPA2 Enterprise with EAP-TLS (Certificate authentication) for my wifi at home. Historically I have used certificates from a public CA for this purpose. This is not best practice since you don’t have control over the certificates that are issued.

Also, I recently bought a new switch capable of 802.1X authentication on all ports. For this purpose I want all my machines (even those without wifi) to have certificates. So I decided to go through the hassle of setting up my own private CA.

Setting up CA

For the basic setup of the CA I followed Jamies excellent guide on setting up a CA. So in this post you can assume that all the basic stuff like folders structure and basic commands are the same. I will only show you the differences needed to have the Root CA key stored on a PKCS11 device like a HSM, Smart Card HSM or a Yubikey. I will even try to follow his topic names so you can follow along.

Configure PKCS11 Engine

I will not discuss the operating system part of getting PKCS11 devices to work in this article. But basically you just need to install some packages, you can read about it here.

First of all we need to configure OpenSSL to talk to your PKCS11 device. This can be done from configuration or interactively on the command line.

From conf:

# At beginning of conf (before everything else)
openssl_conf            = openssl_def

# At end of conf (after everything else)
[openssl_def]
engines = engine_section

[engine_section]
pkcs11 = pkcs11_section

[pkcs11_section]
engine_id = pkcs11
dynamic_path = /usr/local/lib/engines/pkcs11.so
MODULE_PATH = /usr/local/lib/opensc-pkcs11.so
init = 0

From cli:

OpenSSL> engine -t dynamic -pre SO_PATH:/usr/local/lib/engines/pkcs11.so -pre ID:pkcs11 -pre LIST_ADD:1 -pre LOAD -pre MODULE_PATH:/usr/local/lib/opensc-
pkcs11.so

Create the root pair

First of all we need to have a RSA key pair on the PKCS11 device:

# pkcs11-tool --module /usr/local/lib/opensc-pkcs11.so -l --keypairgen --key-type rsa:2048 --label "SSL Root CA"
Using slot 0 with a present token (0x0)
Logging in to "HSM 2 (UserPIN)".
Please enter User PIN:
Key pair generated:
Private Key Object; RSA
  label:      SSL Root CA
  ID:         d15c3e9578a612a658bb14e0e147db4f2279cf19
  Usage:      decrypt, sign, unwrap
Public Key Object; RSA 2048 bits
  label:      SSL Root CA
  ID:         d15c3e9578a612a658bb14e0e147db4f2279cf19
  Usage:      encrypt, verify, wrap

Create the root certificate

I will assume that you have configured pkcs11 in openssl.cnf (otherwise you will have to first run the engine command in openssl interactively before any other command).

# openssl req -config openssl.cnf -new -x509 -days 7300 -sha256 -extensions v3_ca -engine pkcs11 -keyform engine -key 0:d15c3e9578a612a658bb14e0e147db4f2279cf19 -out certs/ca.cert.pem
engine "pkcs11" set.
Enter PKCS#11 token PIN for HSM 2 (UserPIN):
0x8018b6000 07:41:35.523 cannot lock memory, sensitive data may be paged to disk
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [SE]:
State or Province Name []:
Locality Name []:
Organization Name [PeanOrg]:
Organizational Unit Name []:PeanOrg Certificate Authority
Common Name []:PeanOrg Root CA
Email Address []:

Create the intermediate pair

For the intermediate key pair I followed jamies guide. I need frequent access to this CA so I have decided to have the intermediate pair on file instead of HSM.

Create the intermediate certificate

I changed one thing in jamies intermediate/openssl.cnf because I dont see the point of having province set in the CAs

stateOrProvinceName     = optional

To use the Root key stored on pkcs11 to sign the intermediate certificate use this command:

# openssl ca -config openssl.cnf -extensions v3_intermediate_ca -days 3650 -notext -md sha256 -engine pkcs11 -keyform engine -keyfile 0:d15c3e9578a612a658bb14e0e147db4f2279cf19 -in intermediate/csr/intermediate.csr.pem -out intermediate/certs/intermediate.cert.pem
Using configuration from openssl.cnf
engine "pkcs11" set.
Enter PKCS#11 token PIN for HSM 2 (UserPIN):
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4096 (0x1000)
        Validity
            Not Before: Apr  7 05:54:22 2018 GMT
            Not After : Apr  4 05:54:22 2028 GMT
        Subject:
            countryName               = SE
            organizationName          = PeanOrg
            organizationalUnitName    = PeanOrg Certificate Authority
            commonName                = PeanOrg Intermediate CA 1
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                77:9C:07:23:FD:40:E9:5C:7E:30:73:8F:59:28:25:F5:06:43:B4:70
            X509v3 Authority Key Identifier:
                keyid:A4:F2:DE:15:8E:9E:A8:87:B0:95:D4:21:A2:BD:4C:41:02:93:E0:8D

            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
Certificate is to be certified until Apr  4 05:54:22 2028 GMT (3650 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

We now have all we need to sign certificates. Just follow Jamies guide Sign server and client certificates

References

It took me a few hours to get this going because of sort of a lack of documentation on how to use OpenSSL and PKCS11 together, during my efforts I found these resources helpful

https://jamielinux.com/docs/openssl-certificate-authority/index.html
https://github.com/OpenSC/OpenSC/wiki/SmartCardHSM
https://github.com/OpenSC/libp11
https://dennis.silvrback.com/openssl-ca-with-yubikey-neo
https://developers.yubico.com/PIV/Guides/Certificate_authority.html
https://developers.yubico.com/YubiHSM2/Usage_Guides/OpenSSL_with_pkcs11_engine.html

Minimal scp only chroot in FreeBSD

Background

The other day I needed to receive a few files from a friend. And I wanted to provide an easy to use service to do so. Back in the days I did run a FTP service with anonymous login and write acces to a inbox folder (1777).  But these days I have a few more security concerns and wanted to provide a scp drop site, and exactly that. A “scp only chroot”.

Configuration

Giving chrooted ssh access is pretty straight forward in the sshd_config. But to actually get it to work you will have to fiddle around a bit. You will need to figure out exactly what files you will need inside the chroot for the service to function properly.

Create user

Just use your favourite way of creating users in your system.

sshd

The easy part is to configure your sshd_config to chroot the specific user, group or whatever you want to chroot. Since this is probably a one off for me I settled for a single user.

Match User scponly
  ChrootDirectory /home/%u
  X11Forwarding no
  AllowTcpForwarding no

This short configuration will chroot the user scponly to /home/scponly when it connects. You just need to restart sshd in order to these settings to take effect.

Setting up environment

The hardest part was to set up the environment since you will need to make all libs and so on available inside the chroot.

Since this was to be a one off solution I will not go through the hassle of loop/null mounts or stuff like that. I will simply copy the stuff thats actually needed into the chroot. To find out what libs you actually need you can use the tool ldd(1)

# ldd /usr/local/bin/scp 
/usr/local/bin/scp:
	libcrypto.so.8 => /lib/libcrypto.so.8 (0x800a00000)
	libz.so.6 => /lib/libz.so.6 (0x800e69000)
	libutil.so.9 => /lib/libutil.so.9 (0x801082000)
	libldns.so.2 => /usr/local/lib/libldns.so.2 (0x801296000)
	libcrypt.so.5 => /lib/libcrypt.so.5 (0x8014f3000)
	libc.so.7 => /lib/libc.so.7 (0x801712000)

So I just created a few directories in /home/scponly

# cd /home/scponly
# mkdir etc bin lib dev libexec

and then copied these libs to /home/scponly/lib/.

You will also need to have a shell that then can invoke scp. I copied /rescue/sh to /home/scponly/bin/ because it is statically linked and does not depend on any libraries. When all this was done I thought the whole thing was finished but I encountered a few problems.

Problems

The first one was scp complaining about missing ld-elf.so.1. So just like the other libs I just copied it in to /home/sftponly/libexec/.

Secondly scp started to complain about missing /dev/null. And my solution to this was to just create a empty file called null in /home/sftponly/dev/ and chmod to 666. Why it works with a regular file I dont know. If you have any idea please tell me in the comments.

The final problem I faced was an error when invoking scp that said “unknown user 1005”. I did some searches on the web and found several solutions that where Linux specific so no help there. But they involved putting a few more files and libs relevant to nss into the chroot.

What I finally found out was that its the /etc/master.passwd (think of shadow in linux) and its database that where missing. So I just did grep scponly /etc/master.passwd > /home/scponly/etc/master.passwd to copy only the record for scponly to the chroot.

sad:*:1005:1002::0:0:scp only user:/home/scponly:/bin/sh

Finally the actual databased needed to be created with this command

# pwd_mkdb -d /home/scponly/etc/ /home/scponly/etc/master.passwd

Thats it! Now I have a account thats basically all it can do is scp. Yes you can login and get a shell but since there is no other binaries than scp and sh you are pretty limited.

Cheap out of band management/ipmi replacement for home servers

This article describes how to build your own out of band system for your home server.

Background

If you have ever worked professionally with servers you have almost certainly come across some type of out of band management. HP iLO, Dell iDRAC, Huawei iBMC and so on. All more or less based on the IPMI standard. These are basically a small computer that lives independently from your server and give you control over the server even if its turned off for example. You can also virtually mount ISO images and get a console over a web interface and so on. In this way you can even do changes to BIOS settings and so on from a remote location. But probably the most used feature where I work is to simply send power commands to the server like “ipmitool -H <IPMI IP> chassis power off

The problem for home users is that most consumer grade hardware will not have these features since the IPMI is an actual computer that sits on your motherboards (either directly or in some sort of expansion slot). I will now describe what I did to get a similar (but heavily reduced feature set) for my home server.

Hardware

Computer

First of all you need some hardware, this could be any small computer that will fit inside of closely to your existing server. Its also very good if it could get power from the USB port of the actual server, this way you will not have to have any separate power source for your of of band, also this will have ground in the two systems at the same level which will be good later on.

I use a OrangePi Zero and a case which I got off of ebay for about 10USD.

Console

Motherboards these days don’t have a external serial port anymore, but some of them have a internal one which mine luckily had. But to hook up the serial interface of the computer which is a RS-232 interface to the TTL serial interface of the OrangePi I needed to do some conversion or you will damage your Pi, I use a MAX3232 in a package from sparkfun to do this. Just hook upp RX/TX and GND (and 3-5.5V to power it) of both systems to the MAX3232 and you should be good to go. Notice: that if you use the primary serial interface of your Pi you can run into problems if both systems are booting at the same time, because they will interpret each others boot messages as input and will most likely hang the boot.

MX3232

Power control

This is one of the most important and also interesting parts of the project. To have console access to a server is pretty common and I have also described this in another “out of band” article where I use console access for my router.

But for this project I wanted to use the GPIO pins of my OrangePi to control the ATX power switch of my server. After some trial and error I got it to work. I use a transistor as a power switch and then I signal the transistor when I want to power on or off the system. A problem I had early on was that when the server (and then also the Pi) got power it immediately  turned on the server as well. This was a problem for me because of the console “bug” I described above. To work around this I added a couple of transistors to the system, first a 37kΩ resistor in front of the base of the transistor and also a 10kΩ pull-down resistor the get rid of any signal from the GPIO pin when it is not yet configured. The schematics of the thing looks something like this:

Schematics

This was built using:
Adafruit Perma-Proto Quarter-sized Breadboard
TIP120 transistor
a few pin headers and some resistors

and the end result looks something like this:

“Front”
“Back”

The lonely pin is where I connect the GPIO on my OrangePi and to the four other pins I connect the power push button on my case and the pwr+,pwr- pins on the motherboard.

The computer mounted inside my case looks loke this:

OrangePI IPMI

Software

To get the console output working from there server we need to change a few files. You will need something like this in /etc/ttys

ttyu0	"/usr/libexec/getty 3wire"	vt100	on secure

and something like this to /boot/loader.conf

console="comconsole"

I have not yet written any scripts and so on but it’s really easy to use use standard CLI commands in FreeBSD to control the GPIO.

# Setup of the pin
root@orangepi:~ # gpioctl -n 12 pwr
root@orangepi:~ # gpioctl -c pwr out
# Power on (if off)
root@orangepi:~ # gpioctl pwr 1 ; gpioctl pwr 0
# Soft shutdown (if on)
root@orangepi:~ # gpioctl pwr 1 ; gpioctl pwr 0
# Forceful shutdown
root@orangepi:~ # gpioctl pwr 1 ; sleep 5; gpioctl pwr 0

Summing up

This project took me about 3-4 evenings to finish including a lot of trial and error in the beginning and now I have a system where I can actually from a remote location reboot my server. I also get console access early enough to change the options to the kernel or even change kernel before booting. Since I spent only about 25USD in total on this project I’m really satisfied with the result. We will have to see how good it actually works in a real situation, like a kernel panic or whatever.

Strong authentication of OpenSSH hosts using hardware tokens

This article describes how to secure your SSH host keys using hardware tokens.

Background

I have previously written numerous posts about strong user authentication using smart cards, yubikeys and/or OpenSSH certificates. I have also described how you can use OpenSSH certificates to authentcate hosts. But let say your server where compromised in some way just for a few hours and your SSH host keys (and certificates) where stolen. This would allow the attacker to perform MITM attacks or impersonate your server in several different ways. Depending on the application this could be more or less detrimental. If the server have a lot of users this could be used to steal passwords for example.

One way to mitigate this attack is to store your host keys on hardware tokens. This way they cannot be stolen by someone not in physical contact with the server, and you would easily find out if the token was missing.

Configure the token

I will use a Yubikey 4 for this. I have already described how to use yubikeys for client keys, so a more detailed description on how to configure and use Yubikeys for SSH can be found here. You can check that everyting is working like this:

# opensc-tool -l
# Detected readers (pcsc)
Nr.  Card  Features  Name
0    Yes             Yubico Yubikey 4 OTP+U2F+CCID 00 00

What we need is basically a pkcs11 capable device i.e a smart card or “similar” and create a  RSA key pair on it.

Configure sshd and setting up a ssh-agent

Since it would be extremely impractical to enter the token PIN every time a client connects to your secure server, we will need to use a ssh-agent that will keep the pin in memory as long as the agent process is running. We also need to enter the PIN before we leave:

# ssh-agent -a /root/yubikey-agent
setenv SSH_AUTH_SOCK /root/yubikey-agent;
setenv SSH_AGENT_PID 10894;
echo Agent pid 10894;
# setenv SSH_AUTH_SOCK /root/yubikey-agent;
# ssh-add -s /usr/local/lib/opensc-pkcs11.so
Enter passphrase for PKCS#11: 
Card added: /usr/local/lib/opensc-pkcs11.so
# ssh-add -l
2048 SHA256:qBbMpdbUeabLe4PnfjrjPbGPu8zfbkbK+ni4mXOnV24 /usr/local/lib/opensc-pkcs11.so (RSA)

Now you have your RSA key available to the system using the UNIX-domain socket at /root/yubikey-agent.

In a production situation I would put this in a rc-script and enter the PIN at boot-time.

Configure OpenSSH

We need to  create a file containing the public key and also tell sshd to use this key and the newly created socket as a backend for its host keys.

# ssh-keygen -D /usr/local/lib/opensc-pkcs11.so > /etc/ssh/yubikey_host_key.pub
# echo "HostKey /etc/ssh/yubikey_host_key.pub" >> /etc/ssh/sshd_config
# echo "HostKeyAgent /root/yubikey-agent" >> /etc/ssh/sshd_config

Notice that if you have several keys on your token, ssh-keygen will output all of them. Make sure only the correct keys are added to the key file.

Verify the configuration

When you have configured sshd you will need to restart sshd and them we can verify that host keys are actually from the hardware token (in this case the yubikey). A very easy way to do this is to actually try to connect to the server and have a look at what keys it presents to you.

% ssh server
The authenticity of host 'server (172.25.0.15)' can't be established.
RSA key fingerprint is SHA256:qBbMpdbUeabLe4PnfjrjPbGPu8zfbkbK+ni4mXOnV24.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)?

You can now easily verify that this fingerprint IS actually the same as the one ge got when we added the yubikey to the ssh-agent.

This is basically it. If you are going to use this in production you would probably want to add some rc-scripts to automate the setup process as much as possible.

SSH certificates together with hardware keys (smartcard/yubikey)

We have showed how to use SSH certificates and SSH CAs, we have also showed how you can use the yubikey to store you SSH keys. This article will describe how to combine these two features.

First of all you need to have a yubikey set up with some RSA/ECDSA keys. Then find out the public part of you key:

% ssh-keygen -D /usr/local/lib/opensc-pkcs11.so
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCHn4jSqvNLn5NtUxqlAlm1Qj1tlunb0PjBsItmmesquULAM5oqVYmwJ+bmXpDlzgeeIbl1lf3aUTsXCs6My8mkIUwwN+3a5AJ8MA18Bzzx/qIpxe2N/nZ61e03ua5B6SjpfxAAC6i3DPHU6kUSy26sXhfx14y6abWlrwhXaILsTogz2sOganS44au+nSpa35xwMxG5vehkRkYe6vQvvIDFeMhy61DAJGOcGeCbXTfJB9yYwAgOEqTvHDBfbTrHhmnUu82/JV6twY4/tdgrjFxGE3/JsRnoP3lBCoLQR+Uxs3mV7pFelQj/8bZKVsjdzqH9AYWpvIQTJGuwAmyOk61V

This public key should be handed over to your systems administratior (probably yourself in this case) for signing. In return you will get a OpenSSH certificate file that looks sort of a public key but a bit longer.

Now the “tricky” part comes. When you have your public key in ~/.ssh/id_rsa.pub ssh will automatically look for a certificate file called id_rsa-cert.pub. But since we are going to use the smartcard/yubikey to handle our key it will not be visible in ~/.ssh.

First give your certificate a reasonable name like ~/.ssh/yubikey-cert.pub. Then we could tell ssh to combine the two.

Me most basic way is then to just specify the options you need on the command line:

% ssh -o PKCS11Provider=/usr/local/lib/opensc-pkcs11.so -o CertificateFile=~/.ssh/yubikey-cert.pub peter@torus
Enter PIN for 'PIV_II (PIV Card Holder pin)': 
[torus:~] peter>

If you want to use this permanently you can of course put the options in ~/.ssh/config instead it should look something like this:

PKCS11Provider=/usr/local/lib/opensc-pkcs11.so
CertificateFile=~/.ssh/yubikey-cert.pub

A third option if you are using the ssh-agent (like me) you could first add the card to your agent:

% ssh-add -s /usr/local/lib/pkcs11/opensc-pkcs11.so
Enter passphrase for PKCS#11: 
Card added: /usr/local/lib/pkcs11/opensc-pkcs11.so

and the specify the cert-file either on the command line or in ~/.ssh/config

% ssh -o CertificateFile=~/.ssh/yubikey-cert.pub peter@torus
[torus:~] peter>

Now you should be able to combine ssh certificates and yubikeys/smart cards

Securely update wordpress instance

Background

WordPress wants you to have automatic updates turned on for your installation. According to them this is the best way to securely update wordpress. While this is party true because time is key when it comes to web security. If you have patched your installation before anyone tries to exploit the vulnerabilities you might have that’s a good thing. But the problem is that many of these vulnerabilities depend on the web server having write access to your files. And in order to have automatic updates turned on you will have to grant wordpress (the web server) write access to all files that it might want to update.

Securely update wordpress

For a long time wordpress have offered another way of semi automatically updating your wordpress, this depends on you handing over your credentials for file transfers to wordpress. What wordpress does is basically a regular login using ftp or sftp and uploading the new files this way. According to me this is a vast improvement (maybe because I’ve been the victim of wordpress “hacks” that used the possibility to change my files on disk).

But what if you only have sftp/ssh access with key? Or you dont want to give your personal password to wordpress every time you want to do upgrades?

WordPress have a built in solution for this very similar to the one described above. You can actually use ssh keys to do the upgrade.  Create a RSA key pair WITH PASS PHRASE on your web server and store it somewhere safe where only you and the httpd daemon have access to read it.

Then its time to allow this key access to your account, but please restrict access from localhost or the servers own IP. This key should never be allowed to be used from outside. Your .ssh/authorized_keys could look something like this:

[..]
from="127.0.0.1" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC19oGbaEW7NKBQ5vn3auFbUKAasopYYPv03FxhEjbZwhoVTO44BIR0oIdMs1u1v5Y4gDH77ndCI/6fqwJQwc1D0YYH/45wUOEaB4MuPlCxlp7yxE+FyzMspi9mP8HETS+jEfzLIQ01F424yfVweQME9fxeCP0MFO+XK0SuMCk5ibdvaxYwCuRwPFkHcnyKIrDnIgGXv0D8YdC+K/RW/Ghpu9C7Rn2q0pQDbSHj7/xddO7aD+X6DPZfbHS/5ZrJnB+oWf7b9j5FmH8ldBSGBvUr6kplnDr1dKN/98bwRp1FpcxzShAX3q9nj44FwPhKV5JEOw146YJxXXks40ia1da5

Then you can configure this in wp-config.php

define('FTP_PUBKEY','/usr/local/www/ssh/updatekey.pub');
define('FTP_PRIKEY','/usr/local/www/ssh/updatekey');
define('FTP_USER','peter');
define('FTP_HOST','framkant.org');

When doing updates in the future wordpress will ask you for the pass phrase of your ssh key instead of your personal password. In this way the sensitive authentication “material” is never transferred over the internet.

There is still the possibility that malicious code in wordpress or a malicious plugin/theme could steal this key since it have read access to the private key and you give it the pass phrase. But this is still much better than giving it your password.

Scalable access control using OpenSSH Certificates

Background

I’ve been using OpenSSH certificates for some time now. They are very handy if you have a bunch of machines you want to trust, or a bunch of machines that shoud trust you.  It’s very effective to trust just one host CA in order to trust all servers with certificates signed by this CA. Or the other way around, have your personal public key signed by a user CA and then be automatically trusted by all servers that trust this CA. But if you working together with alot of people let say within an organisation this becomes problematic pretty soon. Maybe your frontend people should only have access to webservers and database people to the database servers and so on. The solution to this is the little known flags AuthorizedPrincipalsFile and AuthorizedPrincipalsCommand in sshd_config.

OpenSSH Certificates and principals

Lets start with the certificates and the principals within these certificates. In order to make this work I would suggest to use principals within the certificates that are closely tied to the person using it, their company wide username for example. If different keys have different access leves (lets say because on of them are stored on a physical secure element/smart card) it is good to include this kind of information in the principal, according to some standard you make up. Let say I use yubikeys to store my private keys I could have principals like peter_file and peter_physcial. These are easily parsable and  connected to a physical person.

Please note that there could be other access schemes where role is more important that how the key is stored, then role could be a better option for the principal suffix. But please note that if you burn the role into the certificate that person will need to have the current certificate revoked and have a new one issued if the role is ever changed. I will discuss a better way to handle this later.

Another option is to have multiple principals in the same certificate (ie peter,webmaster,root) but this also gets cumbersome when privileges and roles start to change over time.

AuthorizedPrincipalsFile

One solution to this problem is the configuration option AuthorizedPrincipalsFile in sshd_config. With this option you tell sshd where to look for a list of principals valid for a certain user. I looks something like

AuthorizedPrincipalsFile /etc/ssh/%u_principals

When someone tries to log in as peter sshd will check my certificate for validity and then look for valid principals in the file /etc/ssh/peter_principals. sshd expects this file to contain one valid principal per line and optionally preceded by extra options using the same format as the authorized_keys file. (ie from= and command=). This is flexible enough. I can now give multiple principals (or physical persons) access to a specific account by changing a file. I can also restrict access to certain hosts or create force commands for specific principals.

One use case for this could be a webserver where multiple principals(persons) should be able to use the “webmaster” account but at the same time the test/build system should only be allowed to run a certain commands to publish successful builds. Lets say that the account name is www, then the /etc/ssh/www_principals could look something like this:

peter
erik
from="buildserver.corp.com" command="/bin/publish_website" buildserver
from="guestcomputer.corp.com" guestworker

Please note: If you are really concerned with security, maybe you have given a certificate to a external partner or something, I would suggest to burn the from and command attributes into the certificate. In this way they will never be overridden by some configuration at the server side. The downside is that you will need to produce a new certificate if something changes.

If you have a pretty static setup and/or a decent configuration manager/orchestration tool this could be enough. It gives full flexibility on who should be able to access what, and how. But in the long run it could be tedious to manage all the principal files. This is where the AuthorizedPrincipalsCommand comes in to the picture.

AuthorizedPrincipalsCommand

This works exactly the same as AuthorizedPrincipalsFile but instead of a static file sshd will run a command followed by some options (basically the username that tries to log in) that will generate the principals file dynamically. This gives you a lot of options. Probably the most straight forward one is that you now can have a single ACL file for you whole environment and just let the command read it and produce a host and user specific principal file. One very simple example could look like this:

# Principal(person) user     host
erik                www      webserver.corp.com
peter               www      webserver.corp.com
guestworker         www      webserver.corp.com

peter               db       database.corp.com
erik                db       database.corp.com

Of course this file could be expanded to include more information and more options but this gives a example on how it can be done.

Other backends

But I think the real power in the AuthorizedPrincipalsCommand is that you now can use whatever backend you like and just have the principals command be a wrapper for this backend which could be some Active Directory, LDAP or whatever you might have at your organisation. This makes it possible to use existing infrastructure and still be able to use ssh certificates which I think is a real killer feature in OpenSSH.

If you have some ideas to improve this concept or an questions, please leave a comment.

Running FreeBSD on NanoPI NEO or OrangePI Zero

This article describes how I installed FreeBSD on two different Allwinner based SoC’s.

Background

Recently I bought a NanoPi NEO and a OrangePi Zero from ebay just for fun, HOPING that FreeBSD would run nicely on them. But at first glance things arent that easy. There is no official images for example. But on the other hand there is a image building tool available at FreeBSD’s official github. This article will describe how I got FreeBSD running on these neat ARM computers. Happily this workflow works for both the Neo and the Zero.

Building the image

We need a u-boot image for the NanoPi. In the ports tree there is a lot of diffenret images but not for the nanopi, so what I did was simply to make a copy of the orange-pi-one port and edited it. I basically created my own local port for the NanoPI u-boot. This is what I did:

$ git diff origin/master u-boot-nanopi-neo/Makefile
diff --git a/sysutils/u-boot-nanopi-neo/Makefile b/sysutils/u-boot-nanopi-neo/Makefile
new file mode 100644
index 000000000000..552ba8c1c79f
--- /dev/null
+++ b/sysutils/u-boot-nanopi-neo/Makefile
@@ -0,0 +1,10 @@
+# $FreeBSD: head/sysutils/u-boot-orangepi-one/Makefile 431327 2017-01-12 22:49:08Z imp $
+
+MAINTAINER=    imp@FreeBSD.org
+MASTERDIR=     ${.CURDIR}/../u-boot-master
+
+MODEL=         nanopi-neo
+BOARD_CONFIG=  nanopi_neo_defconfig
+FAMILY=                allwinner
+
+.include "${MASTERDIR}/Makefile"

Amazingly this port compiled and installed without any problems.

Then I fetched the build tool “crochet” from github and created a branch for my changes

$ git clone https://github.com/freebsd/crochet.git
Cloning into 'crochet'...
remote: Counting objects: 5124, done.
remote: Total 5124 (delta 0), reused 0 (delta 0), pack-reused 5124
Receiving objects: 100% (5124/5124), 8.13 MiB | 2.06 MiB/s, done.
Resolving deltas: 100% (2643/2643), done.
$ cd crochet
$ git branch nanopi
$ git checkout nanopi 
Switched to branch 'nanopi'                                                                                                        

To define the NanoPI NEO board I was heavily inspired by the work of Jared McNeill. I basically copied his board definitions with a few changes to suit my setup. Here is the end result:

$ git diff origin/master
diff --git a/board/NanoPi-NEO/overlay/boot/loader.conf b/board/NanoPi-NEO/overlay/boot/loader.conf
new file mode 100644
index 0000000..9c0640f
--- /dev/null
+++ b/board/NanoPi-NEO/overlay/boot/loader.conf
@@ -0,0 +1,3 @@
+# Hack to get loader to find dtb.
+nanopi-neo.dtb_type="dtb"
+nanopi-neo.dtb_load="YES"
diff --git a/board/NanoPi-NEO/overlay/etc/fstab b/board/NanoPi-NEO/overlay/etc/fstab
new file mode 100644
index 0000000..65563ce
--- /dev/null
+++ b/board/NanoPi-NEO/overlay/etc/fstab
@@ -0,0 +1,4 @@
+/dev/mmcsd0s1  /boot/msdos     msdosfs rw,noatime      0 0
+/dev/mmcsd0s2a /               ufs rw,noatime          1 1
+md             /var/log        mfs rw,noatime,-s15m    0 0
+md             /var/tmp        mfs rw,noatime,-s12m    0 0
diff --git a/board/NanoPi-NEO/overlay/etc/rc.conf b/board/NanoPi-NEO/overlay/etc/rc.conf
new file mode 100644
index 0000000..b1ee7a6
--- /dev/null
+++ b/board/NanoPi-NEO/overlay/etc/rc.conf
@@ -0,0 +1,14 @@
+hostname="nanopi-neo"
+ifconfig_awg0="DHCP"
+sshd_enable="YES"
+
+# Nice if you have a network, else annoying.
+ntpd_enable="YES"
+ntpd_sync_on_start="YES"
+
+sendmail_submit_enable="NO"
+sendmail_outbound_enable="NO"
+sendmail_msp_queue_enable="NO"
+
+# CPU frequency scaling
+powerd_enable="YES"
diff --git a/board/NanoPi-NEO/setup.sh b/board/NanoPi-NEO/setup.sh
new file mode 100644
index 0000000..b93c728
--- /dev/null
+++ b/board/NanoPi-NEO/setup.sh
@@ -0,0 +1,34 @@
+KERNCONF=GENERIC
+UBLDR_LOADADDR=0x42000000
+SUNXI_UBOOT="u-boot-nanopi-neo"
+SUNXI_UBOOT_BIN="u-boot.img"
+IMAGE_SIZE=$((1000 * 1000 * 1000))
+TARGET_ARCH=armv6
+
+UBOOT_PATH="/usr/local/share/u-boot/${SUNXI_UBOOT}"
+
+allwinner_partition_image ( ) {
+    echo "Installing U-Boot files"
+    dd if=${UBOOT_PATH}/u-boot-sunxi-with-spl.bin conv=sync of=/dev/${DISK_MD} bs=1024 seek=8
+    dd if=${UBOOT_PATH}/u-boot.img conv=notrunc,sync of=/dev/${DISK_MD} \
+       bs=1024 seek=40
+    disk_partition_mbr
+    disk_fat_create 32m 16 1m
+    disk_ufs_create
+}
+strategy_add $PHASE_PARTITION_LWW allwinner_partition_image
+
+allwinner_check_uboot ( ) {
+    uboot_port_test ${SUNXI_UBOOT} ${SUNXI_UBOOT_BIN}
+}
+strategy_add $PHASE_CHECK allwinner_check_uboot
+
+strategy_add $PHASE_BUILD_OTHER freebsd_ubldr_build UBLDR_LOADADDR=${UBLDR_LOADADDR}
+strategy_add $PHASE_BOOT_INSTALL freebsd_ubldr_copy_ubldr .
+
+# Put the kernel on the FreeBSD UFS partition.
+strategy_add $PHASE_FREEBSD_BOARD_INSTALL board_default_installkernel .
+# overlay/etc/fstab mounts the FAT partition at /boot/msdos
+strategy_add $PHASE_FREEBSD_BOARD_INSTALL mkdir -p boot/msdos
+# ubldr help and config files go on the UFS partition (after boot dir exists)
+strategy_add $PHASE_FREEBSD_BOARD_INSTALL freebsd_ubldr_copy boot
diff --git a/nanopi_config.sh b/nanopi_config.sh
new file mode 100644
index 0000000..acedcf7
--- /dev/null
+++ b/nanopi_config.sh
@@ -0,0 +1,9 @@
+board_setup NanoPi-NEO
+option ImageSize 3900mb # for 4 Gigabyte card
+
+option Growfs
+option Email peter@framkant.org
+option CompressImage
+option User peter
+
+FREEBSD_SRC=/usr/src12

The few things I needed to change from Jared’s board definition was the few ugly lines in loader.conf and also I added conv=sync to the first dd in setup.sh because FreeBSD gave an error otherwise. If someone knows why I need the two lines in loader.conf please tell me.

Support for these boards are added piece by piece in the HEAD branch of FreeBSD. So I checked out a specific source tree with “HEAD” or basically FreeBSD 12-CURRENT.

$ svn co https://svn0.us-west.freebsd.org/base/head /usr/src12
[..]

After this I was ready to build the Image:

$ ./crochet.sh -c nanopi_config.sh
Starting at Sat Jul 1 11:40:10 CEST 2017
Loading configuration from nanopi_config.sh
Board: NanoPi-NEO
Option: ImageSize 3900mb
Option: Growfs
Option: Email peter@framkant.org
Option: CompressImage
Option: User peter
Source version is: 320481
Building FreeBSD version: 12.0
Image name is:
    /usr/local/crochet/work/FreeBSD-armv6-12.0-GENERIC-320481.img
Building FreeBSD version: 12.0
Object files are at: /usr/local/crochet/work/obj/arm.armv6/usr/src12
Found suitable FreeBSD source tree in:
    /usr/src12
Found U-Boot port in:
    /usr/local/share/u-boot/u-boot-nanopi-neo
Building FreeBSD armv6 world at Sat Jul 1 11:40:13 CEST 2017
    (Logging to /usr/local/crochet/work/_.buildworld.armv6.log)
Building FreeBSD armv6-GENERIC kernel at Sat Jul 1 13:04:58 CEST 2017
    (Logging to /usr/local/crochet/work/_.buildkernel.armv6-GENERIC.log)
Building FreeBSD armv6-GENERIC ubldr at Sat Jul 1 13:10:54 CEST 2017
    (Logging to /usr/local/crochet/work/ubldr-armv6-GENERIC/_.ubldr.armv6-GENERIC.build.log)
Creating a 3900MB raw disk image in:
    /usr/local/crochet/work/FreeBSD-armv6-12.0-GENERIC-320481.img
Installing U-Boot files
Installing U-Boot files
417+1 records in
418+0 records out
428032 bytes transferred in 3.867684 secs (110669 bytes/sec)
385+1 records in
386+0 records out
395264 bytes transferred in 3.055042 secs (129381 bytes/sec)
Partitioning the raw disk image with MBR at Sat Jul 1 13:43:51 CEST 2017
gpart create -s MBR md13
md13 created
Creating a 32m FAT partition at Sat Jul 1 13:43:51 CEST 2017 with start block 1m and label BOOT
active set on md13s1
Creating an auto-sized UFS partition at Sat Jul 1 13:43:52 CEST 2017
md13s2 created
/dev/md13s2a: 3685.4MB (7547776 sectors) block size 32768, fragment size 4096
        using 6 cylinder groups of 626.09MB, 20035 blks, 80256 inodes.
super-block backups (for fsck_ffs -b #) at:
 192, 1282432, 2564672, 3846912, 5129152, 6411392
tunefs: soft updates set
Using inode 4 in cg 0 for 4194304 byte journal
tunefs: soft updates journaling set
tunefs: NFSv4 ACLs set
Mounting all file systems:
Mounting FAT partition 1 at /usr/local/crochet/work/_.mount.boot
Mounting UFS partition 1 at /usr/local/crochet/work/_.mount.freebsd
Installing ubldr in /usr/local/crochet/work/_.mount.boot
Installing FreeBSD world at Sat Jul 1 13:43:55 CEST 2017
    Destination: /usr/local/crochet/work/_.mount.freebsd
Overlaying board-specific files from /usr/local/crochet/board/NanoPi-NEO/overlay
1 block
Installing FreeBSD kernel at Sat Jul 1 13:51:34 CEST 2017
    Destination: /usr/local/crochet/work/_.mount.freebsd
Installing all ubldr files in boot
1394 blocks
Adding user peter with password peter
Unmounting /usr/local/crochet/work/_.mount.boot
Unmounting /usr/local/crochet/work/_.mount.freebsd
Releasing md13
Compressing image
DONE.
Completed disk image is in: /usr/local/crochet/work/FreeBSD-armv6-12.0-GENERIC-320481.img

Copy to a suitable memory card using a command such as:
dd if=/usr/local/crochet/work/FreeBSD-armv6-12.0-GENERIC-320481.img of=/dev/da0 bs=1m
(Replace /dev/da0 with the appropriate path for your card reader.)

Finished at Sat Jul 1 13:59:39 CEST 2017

Writing to SD card

After this there was a simple task of writing the image file to a SD card left:

$ ssh torus 'cat /usr/local/crochet/work/FreeBSD-armv6-12.0-GENERIC-320481.img.xz' | unxz | sudo dd of=/dev/rdisk2 bs=1m
0+408362 records in
0+408362 records out
3899999744 bytes transferred in 36.899393 secs (105692788 bytes/sec)

Booting the NanoPI

After writing the image to SD I just inserted the card, network and power to my NanoPI and after a while I got this in the logs of my router:

Jul  1 19:05:02 gw dhcpd: DHCPDISCOVER from 02:81:14:26:0a:0e via bridge0
Jul  1 19:05:03 gw dhcpd: DHCPOFFER on 172.25.0.182 to 02:81:14:26:0a:0e (nanopi-neo) via bridge0
Jul  1 19:05:05 gw dhcpd: Wrote 0 deleted host decls to leases file.
Jul  1 19:05:05 gw dhcpd: Wrote 0 new dynamic host decls to leases file.
Jul  1 19:05:05 gw dhcpd: Wrote 36 leases to leases file.
Jul  1 19:05:05 gw dhcpd: DHCPREQUEST for 172.25.0.182 (172.25.0.1) from 02:81:14:26:0a:0e (nanopi-neo) via bridge0
Jul  1 19:05:05 gw dhcpd: DHCPACK on 172.25.0.182 to 02:81:14:26:0a:0e (nanopi-neo) via bridge0

Now it was a simple task to log in to the device using peter/peter:

$ ssh 172.25.0.182
Password for peter@nanopi:
Last login: Sat Jul  1 17:16:05 2017 from lune.pean.org
FreeBSD 12.0-CURRENT (GENERIC) #0 r320481: Sat Jul  1 13:10:46 CEST 2017

Welcome to FreeBSD!
[..]

Boot messages

$ dmesg 
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #0 r320481: Sat Jul  1 13:10:46 CEST 2017
    peter@torus.pean.org:/usr/local/crochet/work/obj/arm.armv6/usr/src12/sys/GENERIC arm
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
WARNING: WITNESS option enabled, expect reduced performance.
VT: init without driver.
CPU: ARM Cortex-A7 r0p5 (ECO: 0x00000000)
CPU Features: 
  Multiprocessing, Thumb2, Security, Virtualization, Generic Timer, VMSAv7,
  PXN, LPAE, Coherent Walk
Optional instructions: 
  SDIV/UDIV, UMULL, SMULL, SIMD(ext)
LoUU:2 LoC:3 LoUIS:2 
Cache level 1:
 32KB/64B 4-way data cache WB Read-Alloc Write-Alloc
 32KB/32B 2-way instruction cache Read-Alloc
Cache level 2:
 512KB/64B 8-way unified cache WB Read-Alloc Write-Alloc
real memory  = 536870912 (512 MB)
avail memory = 507977728 (484 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
arc4random: no preloaded entropy cache
random: entropy device external interface
kbd0 at kbdmux0
ofwbus0: 
aw_ccu0:  on ofwbus0
clk_fixed0:  on aw_ccu0
clk_fixed1:  on aw_ccu0
clk_fixed2:  on aw_ccu0
aw_gate0:  mem 0x1f01428-0x1f0142b on aw_ccu0
aw_modclk0:  mem 0x1f01454-0x1f01457 on aw_ccu0
simplebus0:  on ofwbus0
aw_ccung0:  mem 0x1c20000-0x1c203ff on simplebus0
aw_reset0:  mem 0x1f014b0-0x1f014b3 on simplebus0
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
regfix0:  on ofwbus0
regfix1:  on ofwbus0
regfix2:  on ofwbus0
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
aw_sid0:  mem 0x1c14000-0x1c143ff on simplebus0
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
awusbphy0:  mem 0x1c19400-0x1c1942b,0x1c1a800-0x1c1a803,0x1c1b800-0x1c1b803,0x1c1c800-0x1c1c803,0x1c1d800-0x1c1d803 on simplebus0
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
gic0:  mem 0x1c81000-0x1c81fff,0x1c82000-0x1c83fff,0x1c84000-0x1c85fff,0x1c86000-0x1c87fff irq 30 on simplebus0
gic0: pn 0x1, arch 0x2, rev 0x1, implementer 0x43b irqs 160
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
gpio0:  mem 0x1c20800-0x1c20bff irq 14,15 on simplebus0
gpiobus0:  on gpio0
gpio1:  mem 0x1f02c00-0x1f02fff irq 34 on simplebus0
gpiobus1:  on gpio1
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
gpioregulator0:  on ofwbus0
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
generic_timer0:  irq 0,1,2,3 on ofwbus0
Timecounter "ARM MPCore Timecounter" frequency 24000000 Hz quality 1000
Event timer "ARM MPCore Eventtimer" frequency 24000000 Hz quality 1000
rtc0:  mem 0x1f00000-0x1f00053 irq 31,32 on simplebus0
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
cpulist0:  on ofwbus0
cpu0:  on cpulist0
cpufreq_dt0:  on cpu0
cpu1:  on cpulist0
cpu2:  on cpulist0
cpu3:  on cpulist0
a31dmac0:  mem 0x1c02000-0x1c02fff irq 4 on simplebus0
a10_mmc0:  mem 0x1c0f000-0x1c0ffff irq 5 on simplebus0
mmc0: <MMC/SD bus> on a10_mmc0
ehci0:  mem 0x1c1d000-0x1c1d0ff irq 12 on simplebus0
usbus0: EHCI version 1.0
usbus0 on ehci0
ohci0:  mem 0x1c1d400-0x1c1d4ff irq 13 on simplebus0
usbus1 on ohci0
gpioc0:  on gpio0
aw_wdog0:  mem 0x1c20ca0-0x1c20cbf irq 20 on simplebus0
uart0:  mem 0x1c28000-0x1c283ff irq 23 on simplebus0
uart0: console (115384,n,8,1)
gpioc1:  on gpio1
awg0:  mem 0x1c30000-0x1c30103,0x1c00030-0x1c00033 irq 35 on simplebus0
miibus0:  on awg0
ukphy0:  PHY 0 on miibus0
ukphy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
ukphy1:  PHY 1 on miibus0
ukphy1:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
awg0: Ethernet address: 02:81:14:26:0a:0e
iichb0:  mem 0x1f02400-0x1f027ff irq 36 on simplebus0
iichb0: could not find clock
device_attach: iichb0 attach returned 2
aw_thermal0:  mem 0x1c25000-0x1c253ff irq 37 on simplebus0
gpioled0:  on ofwbus0
cryptosoft0: 
Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
usbus1: 12Mbps Full Speed USB v1.0
ugen0.1:  at usbus0
uhub0:  on usbus0
ugen1.1:  at usbus1
uhub1:  on usbus1
mmcsd0: 16GB  at mmc0 50.0MHz/4bit/65535-block
Release APs
WARNING: WITNESS option enabled, expect reduced performance.
arc4random: no preloaded entropy cache
Trying to mount root from ufs:/dev/mmcsd0s2a [rw,noatime]...
arc4random: no preloaded entropy cache
uhub1: 1 port with 1 removable, self powered
arc4random: no preloaded entropy cache
uhub0: 1 port with 1 removable, self powered
GEOM_PART: mmcsd0s2 was automatically resized.
  Use `gpart commit mmcsd0s2` to save changes or `gpart undo mmcsd0s2` to revert them.
lock order reversal:
 1st 0xcd331e88 bufwait (bufwait) @ /usr/src12/sys/kern/vfs_bio.c:3539
 2nd 0xc3470000 dirhash (dirhash) @ /usr/src12/sys/ufs/ufs/ufs_dirhash.c:281
stack backtrace:
lock order reversal:
 1st 0xc345d5d4 ufs (ufs) @ /usr/src12/sys/kern/vfs_subr.c:2602
 2nd 0xcd331e88 bufwait (bufwait) @ /usr/src12/sys/ufs/ffs/ffs_vnops.c:280
 3rd 0xc345c934 ufs (ufs) @ /usr/src12/sys/kern/vfs_subr.c:2602
stack backtrace:
random: unblocking device.
awg0: link state changed to DOWN
awg0: link state changed to UP

For now I have only tested that the machine boots with network and the serial console. During my research I found alot of post about problems with USB for example.

NanoPI NEO
NanoPI NEO
OrangePi Zero
OrangePi Zero

Out Of Band Management server

Background
Sometimes I do stupid stuff like editing my firewall rules at home from a remote location and get myself locked out. Sometimes my internet connection is just broken for one reason or the other, this is when you need a out of band channel to your network. You can buy pretty expensive integrated hardware for this with 3G connection and serial consoles and so on, but since this is a project for my home network i decided to build something using a raspberry pi.

Prerequisites
To get this project going I wanted to have a raspberry pi, some sort of wireless connectivity and a serial console to my router.

So I got a Raspberry Pi 2B, this pretty neat case, a old Huawei E1752 from ebay and finally a Linocell Powerbank as a battery backup. For the actual mobile data I got a pre paid SIM card from Telia and got a few GB of data.

Raspberry Pi in case
Raspberry Pi in case

Physical setup and operating system
This setup is very basic and I just put the Pi inside the case and installed FreeBSD using the official image from freebsd.org. I draw power from the powerbank to the pi, and the powerbank is permanently hooked up to power, this way it will run for maybe an hour or so in the event of a power failure.

3G configuration
The reason i got the pretty old E1752 was because it was dirt cheap and also I was absolutley positive it was supported by the u3g driver in FreeBSD.

It is very easy to set up actually, you just put in into a USB port of your Pi and it shows up as three serial interfaces (and maybe some storage device). The first thing you should do is to put the modem in “modem only” mode by sending some AT-codes

# cu -l /dev/cuaU1.0
AT^U2DIAG=0
OK

Then its time to get nostalgic! edit the /etc/ppp/ppp.conf. This was the first time for me since 1998 or something. Of course you will need to figure out some stuff about your 3G provider and make changes accordingly

default:
 set log Phase Chat LCP IPCP CCP tun command
 set device /dev/cuaU1.0
 set timeout 180

telia:
 set speed 115200
 set timeout 0
# set authname wapuser1
# set authkey wap
 set dial "ABORT BUSY TIMEOUT 3 \
        \"\" \
        AT OK-AT-OK \
        AT+CFUN=1 OK-AT-OK \
        AT+CMEE=2 OK-AT-OK \
        AT+CSQ OK \
        AT+CGDCONT=1,\\\"IP\\\",\\\"online.telia.se\\\" OK \
        ATD*99# CONNECT"
 enable dns
 resolv writable
 set ifaddr 10.0.0.1/0 10.0.0.2/0 255.255.255.255 0.0.0.0
 delete! default
 add! default HISADDR

then just test the connection by running “service ppp onestart”

Serial console
My router is located in a small patch cupboard and there was little room for another machine there, so I had to put my OOBM-server somewhere else in my apartment. Luckily I have RJ45 jacks everywhere that are patched to that cupboard so I could very easily run the serial console over the existing CAT6 cables. On the router side I just use a reglar serial cable with DB9 female on one side and a RJ45 male on the other side. On the OOBM-server side I have a simple USB-serial converter followed by a DB9 female to RJ45 female converter. The USB-serial converter shows up in FreeBSD as a regular serial interface like /dev/cuaU0.

RJ45 to DB9
RJ45 to DB9

Out Of Band functionality
Lets put everything together. The first thing i needed to figure out was how to enable the 3g connection remote, but this was pretty simple because the modem can receive sms messages. So I just send some magic/secret sms to the modem that tells it to connect.

Next problem I encountered was that Telia blocks all (?) incoming ports on the mobile connection and since I want to do ssh based administration this was a problem. To work around this problem I went for a solution where the OOBM-server first sets up the PPP connection and then sets up a ssh connection with remote port forwarding to one of my amazon instances. Then I just ssh to the amazon instance on some port and end up on “localhost” on the OOBM-server. On the amazon instance I have created a specific account only used for this purpose that accepts the ssh key used by the OOBM-server.

To put everything together I wrote a small python script that runs every few minutes and checks for valid sms messages on the modem and if it finds such message fires up the PPP connection and then the ssh connection. I will spare you the hazzle of reading my ugly code but here is some pseudo code describing what it does:

m = connect(modem)
if m.send_at("AT") != "OK":
   print "modem is not responsive"
   exit(1)

#Look for valid activation sms
for msg in m.messages():
    if msg.number == "NYNUMBER" && msg.text == "SECRET"
        activate = True
        break

#Try a maximum of three times to set up the connection.
if activate:
    for tries in range(3):
        start_ppp()

        #Check that we can reach internet
        test_connectivity()

        #check that we actually  reach internet via 3G
        verify_route() 
     
        # set up the reverse ssh (ssh -R 31337:localhost:22 remote_server)
        start_ssh()
     
        #Notify me via pushover that connection is up
        send_push("OOBM link up")
 
        sleep(1800)

        m.delete_message(all)
        stop_ssh()
        stop_ppp()
    
        #Notify me via sms that OOBM link is down
        m.send_sms(NUMBER,"OOBM link is down")
        exit(0)

I have omitted the error handling in the pseudo code but I ensure you that the actual script have some.. 😉

The reason i use push messages when the link is up and sms when the link is down is because this modem doesnt have multiplexing and it cant send sms messages while connected.

When the connection is up and running its a simple task to just ssh to remote_server at port 31337 and then login to the OOBM server. From there you can do further ssh connection from the inside of your network or just use the serial console to talk to the router.

% ssh root@remote_server -p31337
root@oobm:~ #
root@oobm:~ # cu -s 57600 -l /dev/cuaU0
Connected

FreeBSD/i386 (gw) (ttyu0)

login:

This is how the setup looks like:

Network diagram
Network diagram

Here is the server installed at its current location

OOBM server on top of power bank
OOBM server on top of power bank