I recently got a new virtual private server from the German provider Contabo. This article explains how I changed the installed system to use encrypted storage. I also wanted not to store the decryption secret anywhere on the system.

Goals:

Encrypt the root partition using cryptsetup and LUKS
Passing the decryption secret using SSH

WARNING: Following this guide might result in complete and irrecoverable data loss! Before proceeding on a live system, make sure to back up all data to be safe! It’s a good idea to test drive this guide using a virtual machine such as qemu or VirtualBox.

The starting setup

I am currently running Debian 11 on my virtual machine and the rescue image is also based on Debian 11.

My system comes with a simple partitioning scheme, where everything is installed on the root partition. Because my use case is simple, this setup suited me well and so I kept it. If you require a more sophisticated scheme, it’s quite easy to create more logical volumes in the LVM setup below.

In my setup the partitioning scheme was the following:

/dev/sda2 was mounted to /boot
/dev/sda3 was mounted to / (root partition)

I will use the above devices in the commands below. If your setup differs, change the commands to use the appropriate names.

I have used Amazon S3 to store the backup of the root filesystem. Setting up an S3 bucket, and a user with proper access is out of the scope of this article.

Prerequisites

It’s best to have the following details noted down before you start:

an AWS user with proper credentials
- AWS_REGION
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
an S3 bucket with read/write permissions for the above user
- S3_BUCKET (ie. my-backup-bucket)
network settings of the host (use ip address and ip route to find out)
- NETWORK_INTERFACE (ie. eth0)
- HOST_IP (ie. 192.168.0.15)
- HOST_NETMASK (ie. 255.255.255.0)
- GATEWAY_IP (ie. 192.168.0.1)
your SSH public key
- SSH_PUBLIC_KEY

I will be referring to these values using their Bash syntax eg. ${AWS_REGION}, etc.

Booting into the rescue environment

The Contabo VPS provisioning process is automated and very straightforward, but it doesn’t support interactive system installation. What I can do instead, is run the system with the Debian rescue image that allows me to change the root partition.

The following guide assumes that you are booted into your system using a rescue image such as the Debian LiveCD and you have a root console open.

Install and configure AWS CLI

Because I’m using S3 to store the contents of the root filesystem while I change the partitions, I need the aws command line tool. For simplicity’s sake I chose to use the standalone bundle available here.

I installed the tool with these commands:

apt-get update
apt-get install -y curl unzip
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
./aws/install
aws configure set region ${AWS_REGION}
aws configure set output json
aws configure set aws_access_key_id ${AWS_ACCESS_KEY_ID}
aws configure set aws_secret_access_key ${AWS_SECRET_ACCESS_KEY}

Back up contents of the root partition

In this section, I mount the root partition and create a backup file using tar and then upload the file to S3 using aws. Last, I unmount the partition to prepare for encryption.

Because the LiveCD environment is using a ramdisk, it has limited storage capacity available. For this reason, it’s important to create the backup file on the hard disk itself, so you don’t run into problems if the backup file is large.

mkdir -p /mnt/root
mount /dev/sda3 /mnt/root/
cd /mnt/root/
tar cfzp root.tar.gz --exclude=root.tar.gz *
aws s3api put-object --bucket ${S3_BUCKET} --key root.tar.gz --body root.tar.gz 
cd /
umount /mnt/root/

Create the encrypted super-partition

The next step is to change the existing root partition to become an encrypted “super-partition”. The data partitions will be created inside this super-partition and therefore be encrypted themselves.

To encrypt the partition the cryptsetup tool will ask for a password. Be sure to use a strong password and keep it somewhere safe and durable! You will need it every time you reboot the system, otherwise your data is likely lost.

apt-get install -y cryptsetup
cryptsetup luksFormat /dev/sda3
cryptsetup luksOpen /dev/sda3 sda3_crypt

Create encrypted data partitions using LVM

In this section, I am using LVM on the encrypted partition to define logical volumes. In my case it’s a single root volume, but feel free to create as many logical volumes as needed. Because they are in the encrypted partition, all data is stored using encryption.

Here, I create a physical volume, a volume group and a logical volume, and finally format it to ext4. The benefit of ext4 is that it supports dynamic resizing, and resize and add more partitions using LVM on a running system. Other filesystems work fine too, but may not have this feature.

Caveat: make sure to install the tools package for the filesystem type that you chose before you reboot your system!

apt-get install -y lvm2
pvcreate /dev/mapper/sda3_crypt
vgcreate main-vg /dev/mapper/sda3_crypt
lvcreate main-vg -l 100%FREE -n root /dev/mapper/sda3_crypt
mkfs.ext4 /dev/mapper/main--vg-root
mount /dev/mapper/main--vg-root /mnt/root/

Your new root partition device is now /dev/mapper/main--vg-root.

Restore contents of the root partition

I am finally ready to restore the files on the root filesystem, with the following commands:

aws s3api get-object --bucket ${S3_BUCKET} --key root.tar.gz
tar xfzp root.tar.gz
rm  root.tar.gz

Chroot into the system

In this section we are emulating a running system environment by chrooting into it. This is important because in the sections below, we are installing packages and configuration that are required to boot without problems. These tools often use runtime information gathered from files to configure themselves.

mount -t proc none /mnt/root/proc
mount -t sysfs none /mnt/root/sys
mount -t devtmpfs none /mnt/root/dev
mount /dev/sda2 /mnt/root/boot
mkdir -p /mnt/root/run/resolvconf/
cp /run/resolvconf/resolv.conf /mnt/root/run/resolvconf/
chroot /mnt/root /bin/bash
source /etc/profile

Update fstab and crypttab

Edit your /etc/fstab file to change the root partition to the newly created one:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/mapper/main--vg-root                 /               ext4    errors=remount-ro        0       0
# /boot was on /dev/sda1 during installation
UUID=b0adef47-4a54-4b1e-a748-45f0c7c936b3 /boot           ext4    defaults,noatime,noatime      0       0

The blkid command prints the UUIDs of your current partitions:

/dev/sda1: PARTLABEL="primary" PARTUUID="ad24c9bb-c374-415c-b2e0-6ac0e8866120"
/dev/sda2: UUID="b0adef47-4a54-4b1e-a748-45f0c7c936b3" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="primary" PARTUUID="65289e1e-2249-447a-b201-74d2ce58c0cb"
/dev/sda3: UUID="1d09e04c-518e-49c4-9481-fab5fd98913f" TYPE="crypto_LUKS" PARTUUID="400559f6-f0ad-274a-933f-468a9d01637a"
/dev/mapper/sda3_crypt: UUID="Vne23e-0vHg-XmGU-eJQR-ghy5-PQTL-Myc0uO" TYPE="LVM2_member"
/dev/mapper/main--vg-root: UUID="fe8df4b8-5b1a-43e7-8efb-a46904a0137c" BLOCK_SIZE="4096" TYPE="ext4"

Add a line like this to /etc/crypttab with the correct UUID from the command above. The UUID to use is of the unencrypted partition, in this case from the line starting with /dev/sda3.

sda3_crypt UUID=1d09e04c-518e-49c4-9481-fab5fd98913f none luks,discard

This is also a good time to install any filesystem packages if you chose to go with a filesystem other than ext4.

For example, if you used XFS, you would install the relevant package like this:

apt-get install -y xfsprogs

Install dropbear-initramfs to decrypt the root partition via SSH

It’s now time to set the system up to receive the partition decryption key over an SSH connection when the system is rebooted. Without this, we would either need physical access to the machine or a remote console, or worst to store the secret on the machine itself (which defeats the purpose of encryption).

We do this by embedding a small SSH server into the kernel. When the system is rebooted, this will enable you to SSH into the host and enter the password interactively from the comfort of your own device.

We install the required packages, apply configuration and install the server’s own SSH host keys into the kernel as well. Finally the initramfs is updated to make sure that all required pieces are embedded in the kernel at boot time.

The SSH_PUBLIC_KEY is a public key you are going to use to connect to the host to enter the decryption key.

apt-get install -y dropbear-initramfs cryptsetup-initramfs
sed -i -e 's/#DROPBEAR_OPTIONS.*/DROPBEAR_OPTIONS="-RFEsjk -c /bin/cryptroot-unlock"/' /etc/dropbear-initramfs/config

echo "${SSH_PUBLIC_KEY}" > /etc/dropbear-initramfs/authorized_keys
chmod 0600 /etc/dropbear-initramfs/authorized_keys
ssh-keygen -m PEM -p -f /etc/ssh/ssh_host_ecdsa_key
ssh-keygen -m PEM -p -f /etc/ssh/ssh_host_rsa_key
/usr/lib/dropbear/dropbearconvert openssh dropbear /etc/ssh/ssh_host_rsa_key /etc/dropbear-initramfs/dropbear_rsa_host_key
/usr/lib/dropbear/dropbearconvert openssh dropbear /etc/ssh/ssh_host_ecdsa_key /etc/dropbear-initramfs/dropbear_ecdsa_host_key
echo 'IP="${HOST_IP}::${GATEWAY_IP}:${HOST_NETMASK}::${NETWORK_INTERFACE}:none"' > /etc/initramfs-tools/conf.d/ip
update-initramfs -k all -u

Watch for warnings in the output of the update-initramfs command and fix any issues that occur at this stage. Otherwise, you might end up with an unbootable system and you have to repeat some of the process explained here.

Update grub

Finally, we have to update our boot configuration and reboot the system.

update-grub
exit
umount /mnt/root/{proc,sys,dev,boot}
umount /mnt/root/
shutdown -r now

When the system restarts, it will halt to prompt for the decryption key you created above. You can use SSH to log onto the system and supply this key.

I hope you find this guide useful, and feel free to send me feedback!

Credits

I used the following resources to create this guide:

Search This Blog

Thoughts about code

Disk encryption with SSH remote unlocking on Debian 11