SSH to decrypt encrypted LVM during headless server boot?

Question

When I installed Ubuntu 10.04 and, now, 10.10, I was offered the option of enabling "encrypted LVM" for my hard drive. After choosing that option, I am prompted for my password during boot to decrypt the LVM.

Now, I am thinking about setting up a headless server that runs Linux (not necessarily Ubuntu), but I am worried that since the server is headless I won't be able to decrypt it during startup. Would I be able to SSH in during boot to enter my password for the encrypted LVM? If so how do I set it up? Or is there another solution? Again this question is NOT specific to Ubuntu. Thanks.

Also see: zless /usr/share/doc/cryptsetup/README.remote.gz — 0xC0000022L, Jun 02 '14 at 12:41
I think @Nate 's answer should be the accepted one: it (essentially, since an edit is required to reflect changes in the linked blog) uses public keys throughout instead of private ones. — Jonathan Y., Oct 17 '15 at 17:33

score 32 · Accepted Answer · edited May 23 '23 at 11:34

For newer versions of Ubuntu, for example, 14.04, I found a combination of @dragly and these blogposts' answers very helpful. To paraphrase:

(On server) Install Dropbear
```
sudo apt-get install dropbear
```
(On server) Copy and assign permissions for root public/private key login
```
sudo cp /etc/initramfs-tools/root/.ssh/id_rsa ~/.
sudo chown user:user ~/id_rsa
```
Remember to change user to your username on the server.

(On client) Fetch private key from server

scp user@remote.server:~/id_rsa ~/.ssh/id_rsa_dropbear

(On client) Add an entry to ssh config

Host parkia
    Hostname 192.168.11.111
    User root
    UserKnownHostsFile ~/.ssh/know_hosts.initramfs
    IdentityFile ~/.ssh/id_rsa_dropbear

Remember to change parkia to whatever you'd like to type ssh my-box to be.

(On server) Create this file at /etc/initramfs-tools/hooks/crypt_unlock.sh

(On server) Make that file executable

sudo chmod +x /etc/initramfs-tools/hooks/crypt_unlock.sh

Update the initramfs
```
sudo update-initramfs -u
```
Disable the dropbear service on boot so openssh is used after partition is decrypted
```
sudo update-rc.d dropbear disable
```

You're done. Try it out. Check the blog post linked above for instructions about how to configure the server with a static IP address if that is something you'd need to do.

The linked blog has added a reference to appending a client's public key to the server's /etc/initramfs-tools/root/.ssh/authorized_keys, even if it still goes through copying Dropbear's private key, which one can completely disregard. Following the rest of the instructions works for me, which means this should be the accepted answer (once it reflects that change), because it only uses public keys. — Jonathan Y., Oct 17 '15 at 17:31

score 24 · Answer 2 · edited May 23 '23 at 11:38

A guide to do such a setup with BusyBox and Dropbear is shown in this blog post. early-ssh didn't work for me and is apparently not needed any more.

I have summarized what you need to do in the following. For more details, have a look at the post above:

Install BusyBox and Dropbear on your server
```
sudo apt-get install dropbear busybox
```
Update your initramfs on the server
```
sudo update-initramfs -u
```
Copy the private key generated by dropbear to your client machine. You may have to copy this to a new dir and change ownership to do this. On your server do the following:
```
sudo cp /etc/initramfs-tools/root/.ssh/id_rsa ~/.
sudo chown user:user ~/id_rsa
```
Remember to replace user with your username. Password logins don't seem to work.
Now you may transfer the private key with scp by calling the following on your client:
```
scp user@remote.server:~/id_rsa ~/.ssh/id_rsa_dropbear
```
Set up your client's ~/.ssh/config file for easy login. Open it up with a text editor and add the following:
```
Host myremoteserver
    HostName my.remote.server
    User root
    UserKnownHostsFile ~/.ssh/known_hosts.initramfs
    IdentityFile ~/.ssh/id_rsa_dropbear
```
Change the Host to whatever you like and HostName to the name of your server. Let the user be root. It appears to be the only accepted user in Dropbear. Save and close the file.
Restart your server and wait for the passphrase prompt. Give Dropbear a few seconds to detect and set up its internet connection. Connect to your server with the following command on your client:

    ssh myremoteserver # or any name you chose

When logged in, issue the following command on your server. See the blog post for details:
```
pid=`ps | grep "/scripts/local-top/cryptroot" | cut -d " " -f 3`
kill -9 $pid
sleep 35
/scripts/local-top/cryptroot
pid=`ps | grep "/bin/sh" | cut -d " " -f 3`
kill -9 $pid;
```
It will take some time (30 seconds) before you get to type your passphrase. Type it in when prompted.
Close the connection by typing
```
exit
```
Your server should now have unlocked its encrypted hard drive and boot as normal.

(A huge thanks to the original author of the blog post!)

I don't quite understand the reason for moving around with private keys. It should be sufficient to copy the public key of you on the client machine to the server as authorized key for the root server, right? — gertvdijk, Dec 02 '12 at 18:11
I'm sure it's possible to create the key pair on the client machine and only move the public key from there to the server, but if I recall correctly I think there were some problems finding a format that BusyBox would accept. So reusing the keys that were already on the server was the only option I got working. — dragly, Dec 04 '12 at 09:04
Any idea what I should do to get this working on Arch Linux? — Gerharddc, Feb 09 '14 at 06:17
@Gerhman Check out the dropbear_initrd_encrypt package in the AUR for early-ssh support on Archlinux. — Caleb, Feb 03 '15 at 10:27
@Gerhman archwiki page: Remote unlocking of the root or other partition Haven't done it yet, but it looks interesting. Will have to check it out :) — hanetzer, Mar 07 '15 at 20:47

score 18 · Answer 3 · answered Dec 20 '10 at 16:17

18

I think early-ssh provides what you're searching for:

Early-ssh is a simple initramfs hook, which installs Dropbear SSH server into  
your initramfs, and starts it at boottime, so you will be able to do a lot of  
things remotely over SSH, before your root partition gets mounted, for example:

* unlocking LUKS encrypted crypto devices - 
  even your root can be an encrypted filesystem
* assembling/altering RAID arrays (mdadm)
* checking the root filesystem in read-write mode, 
  taking action in case of errors
* and so on...

There is already a .deb package available, so you're probably fine with Ubuntu.

answered Dec 20 '10 at 16:17

wag

35,944
12
67
51

Looks like this is exactly what I am looking for, thanks! – hpy Dec 22 '10 at 23:36
3

Do you have a link to a good tutorial or howto? I am stuck with early-ssh on my debian squeeze box. – Mar 14 '11 at 11:31
1

Yes a tutorial would be great. – hpy Apr 29 '11 at 14:09
http://roosbertl.blogspot.de/2012/12/centos6-disk-encryption-with-remote.html – a coder Jun 24 '15 at 15:25

score 16 · Answer 4 · edited Apr 13 '17 at 12:36

Have a look at the cryptsetup readme for this in /usr/share/doc/cryptsetup/README.remote.gz (Ubuntu package cryptsetup). In there is a full guide to accomplish this. It is similar to dragly's answer, but I think this is a bit more elegant. (Dropbear formatted keys, passing the passphrase via a FIFO rather than a fragile shell script, etc.)

unlocking rootfs via ssh login in initramfs

You can unlock your rootfs on bootup from remote, using ssh to log in to the booting system while it's running with the initramfs mounted.

Setup

For remote unlocking to work, the following packages have to be installed before building the initramfs: dropbear busybox

The file /etc/initramfs-tools/initramfs.conf holds the configuration options used when building the initramfs. It should contain BUSYBOX=y (this is set as the default when the busybox package is installed) to have busybox installed into the initramfs, and should not contain DROPBEAR=n, which would disable installation of dropbear to initramfs. If set to DROPBEAR=y, dropbear will be installed in any case; if DROPBEAR isn't set at all, then dropbear will only be installed in case of an existing cryptroot setup.

The host keys used for the initramfs are dropbear_dss_host_key and dropbear_rsa_host_key, both located in /etc/initramfs-tools/etc/dropbear/. If they do not exist when the initramfs is compiled, they will be created automatically. Following are the commands to create them manually:
dropbearkey -t dss -f /etc/initramfs-tools/etc/dropbear/dropbear_dss_host_key
dropbearkey -t rsa -f /etc/initramfs-tools/etc/dropbear/dropbear_rsa_host_key
As the initramfs will not be encrypted, publickey authentication is assumed. The key(s) used for that will be taken from /etc/initramfs-tools/root/.ssh/authorized_keys. If this file doesn't exist when the initramfs is compiled, it will be created and /etc/initramfs-tools/root/.ssh/id_rsa.pub will be added to it. If the latter file doesn't exist either, it will be generated automatically - you will find the matching private key which you will later need to log in to the initramfs under /etc/initramfs-tools/root/.ssh/id_rsa (or id_rsa.dropbear in case you need it in dropbear format). Following are the commands to do the respective steps manually:

To create a key (in dropbear format):
dropbearkey -t rsa -f /etc/initramfs-tools/root/.ssh/id_rsa.dropbear
To convert the key from dropbear format to openssh format:
/usr/lib/dropbear/dropbearconvert dropbear openssh \
    /etc/initramfs-tools/root/.ssh/id_rsa.dropbear \
    /etc/initramfs-tools/root/.ssh/id_rsa
To extract the public key:
dropbearkey -y -f /etc/initramfs-tools/root/.ssh/id_rsa.dropbear | \
    grep "^ssh-rsa " > /etc/initramfs-tools/root/.ssh/id_rsa.pub
To add the public key to the authorized_keys file:
cat /etc/initramfs-tools/root/.ssh/id_rsa.pub >> /etc/initramfs-tools/root/.ssh/authorized_keys
In case you want some interface to get configured using dhcp, setting DEVICE= in /etc/initramfs-tools/initramfs.conf should be sufficient. The initramfs should also honour the ip= kernel parameter. In case you use grub, you probably might want to set it in /boot/grub/menu.lst, either in the '# kopt=' line or appended to specific 'kernel' line(s). The ip= kernel parameter is documented in Documentation/nfsroot.txt in the kernel source tree.

Issues

Don't forget to run update-initramfs when you changed the config to make it effective!

Collecting enough entropy for the ssh daemon sometimes seems to be an issue. Startup of the ssh daemon might be delayed until enough entropy has been retrieved. This is non-blocking for the startup process, so when you are at the console you won't have to wait for the sshd to complete its startup.

Unlocking procedure

To unlock from remote, you could do something like this:
ssh -o "UserKnownHostsFile=~/.ssh/known_hosts.initramfs" \
    -i "~/id_rsa.initramfs" root@initramfshost.example.com \
    "echo -ne \"secret\" >/lib/cryptsetup/passfifo"
This example assumes that you have an extra known_hosts file "~/.ssh/known_hosts.initramfs" which holds the cryptroot system's host-key, that you have a file "~/id_rsa.initramfs" which holds the authorized-key for the cryptroot system, that the cryptroot system's name is "initramfshost.example.com", and that the cryptroot passphrase is "secret"

-- <debian@x.ray.net>, Wed, 30 Sep 2009

Thanks to jap for pointing this out to me on a different channel.

This seems like a much better idea (being described in the offical docs and everything) than hackish ps-grepping. As a side note on the unlocking procedure provided though, one might want to be cautious about typing the passphrase directly on the command line as it will most likely end up in a shell history file somewhere. A possible solution is creating a tiny wrapper script prompting for the passphrase using read -s -p. — joelpet, Feb 21 '15 at 22:55
note that there are issues with that approach in recent Ubuntu versions, i.e. https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/595648 — Frederick Nord, Apr 09 '15 at 06:37

Teddy · Answer 5 · 2013-10-26T12:47:27.773

6

If you want to be able to boot unattended as well as remotely, you should also look at Mandos (which I and others have written):

Mandos is a system for allowing servers with encrypted root file systems to reboot unattended and/or remotely. See the intro manual page file for more information, including an FAQ list.

In short, the booting server gets the password over the network, in a secure fashion. See the README for details.

edited Oct 26 '13 at 12:47

answered Apr 28 '11 at 11:08

Teddy

1,555
10
13

Thank you for your contributions, but note that 100% of your posts being almost-identical mentions of your product is borderline behavior, and you run a risk of being considered spam (I'd have flagged your posts if Mandos wasn't free software or you didn't have a history of non-Mandos posts on other sites). – Gilles 'SO- stop being evil' Apr 28 '11 at 21:24
@Gilles: Done now. – Teddy May 03 '11 at 11:54

score 4 · Answer 6 · answered Dec 21 '17 at 00:56

On debian 9 (stable), this solution was outdated. During the installation, I get a warning about dropbear: WARNING: Invalid authorized_keys file, remote unlocking of cryptroot via SSH won't work!, and I was not able to find the needed keys. This method is by the way really simple, and was explained to me on the great #debian channel (thank you again):

First make sure that busybox, dropbear and dropbear-initramfs are installed

sudo apt install busybox dropbear*

then add your public key (most of the time ~/.ssh/id_rsa.pub) in the file /etc/dropbear-initramfs/authorized_keys.

Update then initramfs to take into account the changes: : update-initramfs -u

That's all!

Note, if you want to avoid to have clash between the keys between dropbear and openssh (they share the same ip, but use a different key), you may want to put in your client ~/.ssh/config something like that:

Host myserver_luks_unlock
     User root
     Hostname <myserver>
     # The next line is useful to avoid ssh conflict with IP
     HostKeyAlias <myserver>_luks_unlock
     Port 22
     PreferredAuthentications publickey
     IdentityFile ~/.ssh/id_rsa

Then, you just connect using:

ssh myserver_luks_unlock

and once you get a prompt, type as suggested by the busybox prompt :

cryptroot-unlock

and type your password.

Enjoy!

Does this need to be User root ? Ideally I'd want to disable remote login for root. — Jonathan Cross, Apr 12 '20 at 11:28
I'm sorry I don't remember (and I'm not using it right now so can't really test). I'd say no like that, but not sure. But anyway, it's not hard to try I guess, let us know if you try. — tobiasBora, Apr 12 '20 at 13:09

score 3 · Answer 7 · answered Jul 17 '15 at 17:07

3

On arch linux, there is an AUR package dropbear_initrd_encrypt that does what you want out of the box. Works pretty well for wired interfaces. I had to hack it up slightly for wireless.

answered Jul 17 '15 at 17:07

user3188445

5,257

ceremcem · Answer 8 · 2017-08-31T05:43:26.903

Unfortunately, none of above answers worked for me. Moreover, copying a private key from server seems paradoxical.

Anyhow, following instructions worked out:

Boot your SERVER by connecting and unlocking the encrypted partition via your CLIENT

Install mandatory packages (on SERVER)

apt-get install dropbear initramfs-tools busybox

Append your desired public keys into the SERVER's authorized_keys file

Just copy and paste your public key(s) into /etc/dropbear-initramfs/authorized_keys on SERVER

Create the unlock script

Create following script in /etc/initramfs-tools/hooks/crypt_unlock.sh

#!/bin/sh

PREREQ="dropbear"

prereqs() {
  echo "$PREREQ"
}

case "$1" in
  prereqs)
    prereqs
    exit 0
  ;;
esac

. "${CONFDIR}/initramfs.conf"
. /usr/share/initramfs-tools/hook-functions

if [ "${DROPBEAR}" != "n" ] && [ -r "/etc/crypttab" ] ; then
cat > "${DESTDIR}/bin/unlock" << EOF
#!/bin/sh
if PATH=/lib/unlock:/bin:/sbin /scripts/local-top/cryptroot; then
kill \`ps | grep cryptroot | grep -v "grep" | awk '{print \$1}'\`
# following line kill the remote shell right after the passphrase has
# been entered.
kill -9 \`ps | grep "\-sh" | grep -v "grep" | awk '{print \$1}'\`
exit 0
fi
exit 1
EOF

  chmod 755 "${DESTDIR}/bin/unlock"

  mkdir -p "${DESTDIR}/lib/unlock"
cat > "${DESTDIR}/lib/unlock/plymouth" << EOF
#!/bin/sh
[ "\$1" == "--ping" ] && exit 1
/bin/plymouth "\$@"
EOF

  chmod 755 "${DESTDIR}/lib/unlock/plymouth"

  echo To unlock root-partition run "unlock" >> ${DESTDIR}/etc/motd

fi

Make it executable:

chmod +x /etc/initramfs-tools/hooks/crypt_unlock.sh

Create a static IP (or skip this step to use DHCP)

Edit /etc/initramfs-tools/initramfs.conf to add (or change) the line:

#format [host ip]::[gateway ip]:[netmask]:[hostname]:[device]:[autoconf]
#([hostname] can be omitted)
IP=192.168.1.254::192.168.1.1:255.255.255.0::eth0:off

Update initialramfs

update-initramfs -u

Disable the dropbear service on boot so openssh is used after partition is decrypted

sudo update-rc.d dropbear disable

Testing

Reboot your server
Connect to your server via ssh root@192.168.1.254 [-i ~/.ssh/id_rsa]

score 2 · Answer 9 · answered Apr 29 '11 at 12:41

2

Headless server? If it has a serial port, use it.

GRUB can be configured to work over the serial port. Your kernel can also be configured use the serial port for outputting the initial boot messages, inputting the password to unlock your drives, and logging in. (If your server supports serial BIOS, enable that as well. Then you'll never have to connect a monitor to the machine at all).

Always a good idea to have a "non-network" way of getting in a headless server.

answered Apr 29 '11 at 12:41

LawrenceC

10,992

1

Great point! Though, "non-network" ways of getting in a headless server are mostly (only) relevant if the physical proximity between client/server is close; unless I'm overlooking some other possibility/feature of a Serial connection. – ILMostro_7 Feb 28 '16 at 06:53

xolox · Answer 10 · 2020-08-07T13:15:18.327

I've been using the technique explained by others on this page (SSH in the initramfs with a kernel IP parameter to configure networking) for quite a few years now to remotely unlock headless Ubuntu Linux servers (12.02, 14.04, 16.04 and 18.04).

I even went so far as to develop a Python program (unlock-remote-system) that does the actual unlocking for me, because the process of doing this manually felt a bit fragile and I started to dread rebooting my servers, so in the spirit of "if it hurts it's worth automating" I encoded my knowledge in Python (and this has indeed made it much easier to do regular reboots to apply security updates).

Since then I decided to also share my personal notes on Remote root disk encryption with the world. The linked page contains quite a few details about the procedure (also some hints that are not mentioned here) and I intend to keep it up to date.

ypid · Answer 11 · 2017-04-23T07:49:00.973

1

I wrote an Ansible role which does this for you. Just get the role debops-contrib.dropbear_initramfs and run it. Refer to the role’s documentation for details.

edited Apr 23 '17 at 07:49

answered Mar 01 '15 at 17:46

ypid

181

SSH to decrypt encrypted LVM during headless server boot?

11 Answers11

unlocking rootfs via ssh login in initramfs

Setup

Issues

Unlocking procedure