Automounting LUKS encrypted external usb when device inserted without error or hanging on boot if not present

Question

Scenario

Have an external device, encrypted with LUKS, automatically mount when inserted and not cause any issues if not present on boot.

Most questions on this topic refer to mounting on boot - this is NOT what I am trying to do

Additionally, the mountpoint should be protected from accidental writes when the usb device is not present and I can do this with chattr +i <mountpoint>

Problem

1. When the external device is NOT mounted, any attempt to access the mountpoint will hang until timeout - this can cause system instability.

2. When the device is inserted, the passphrase is prompted for and device unlocked, but not mounted. I then have to mount the device manually with mount /mnt/backup or mount /dev/mapper/fit which appears to correctly use the fstab settings.

Configuration

# /etc/crypttab
fit   UUID=xxxxxxxx-xxxxx-xxxxxxxx-xxxxxxxxxx none luks,noauto

# /etc/fstab
/dev/mapper/fit    /mnt/backup   ext4    noatime,user,noauto,x-systemd.automount,x-systemd.device-timeout=5ms,x-systemd.mount-timeout=100ms    0   0

I happen to have a Samsung Fit usb flash drive if anyone was wondering about the name

Accessing the mountpoint when the external device is NOT present

# Accessing the mountpoint when the drive is NOT plugged in
andy@pop-os:mnt$ ll
ls: cannot access 'backup': No such device
total 0
d????????? ? ? ? ?            ? backup/
andy@pop-os:mnt$ mountpoint /mnt/backup
/mnt/backup is a mountpoint

# journalctl
Jan 21 16:33:34 pop-os systemd[1]: mnt-backup.automount: Got automount request for /mnt/backup, triggered by 5192 (ls)
Jan 21 16:33:34 pop-os systemd[1]: dev-mapper-fit.device: Job dev-mapper-fit.device/start timed out.
Jan 21 16:33:34 pop-os systemd[1]: Timed out waiting for device /dev/mapper/fit.
Jan 21 16:33:34 pop-os systemd[1]: Dependency failed for /mnt/backup.
Jan 21 16:33:34 pop-os systemd[1]: mnt-backup.mount: Job mnt-backup.mount/start failed with result 'dependency'.
Jan 21 16:33:34 pop-os systemd[1]: dev-mapper-fit.device: Job dev-mapper-fit.device/start failed with result 'timeout'.

Normal Removal Processes

# Absolute path is required if not using sudo (`user` was set in fstab)
andy@pop-os:mnt$ umount /mnt/backup
andy@pop-os:mnt$ sudo cryptsetup close fit
andy@pop-os:mnt$ sudo eject /dev/sdx

Temporary Resolution

I can unmount the mountpoint (even though no device is mounted anyway) to temporarly fix this issue but it will come back after a system reboot or after mounting/ummounting the device again.

andy@pop-os:mnt$ sudo umount backup
andy@pop-os:mnt$ ll
total 4.0K
drwxr-xr-x 2 root root 4.0K Jan 19 10:16 backup/
andy@pop-os:mnt$ lsattr
----i---------e------- ./backup
andy@pop-os:~$ mountpoint /mnt/backup
/mnt/backup is not a mountpoint

If I use noauto and don't include x-systemd.automount then I avoid the problem of the directory being a mountpoint on boot when no device is present, but I also don't get automounting - although the device still unlocks.

Mounting the external device

Passphrase is prompted for and device unlocked

andy@pop-os:mnt$ lsblk -f
NAME            FSTYPE      FSVER    LABEL     UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sdb             crypto_LUKS 2                  xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx                  
└─fit           ext4        1.0                yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy
andy@pop-os:mnt$ ll backup/
ls: cannot access 'backup/': No such device

Note that MOUNTPOINTS is empty

# journalctl -f
Jan 21 17:17:26 pop-os kernel: usb 6-2: new SuperSpeed USB device number 2 using xhci_hcd
Jan 21 17:17:26 pop-os kernel: usb 6-2: New USB device found, idVendor=090c, idProduct=1000, bcdDevice=11.00
Jan 21 17:17:26 pop-os kernel: usb 6-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan 21 17:17:26 pop-os kernel: usb 6-2: Product: Flash Drive FIT
Jan 21 17:17:26 pop-os kernel: usb 6-2: Manufacturer: Samsung
Jan 21 17:17:26 pop-os kernel: usb 6-2: SerialNumber: 0123456789123
Jan 21 17:17:26 pop-os kernel: usb-storage 6-2:1.0: USB Mass Storage device detected
Jan 21 17:17:26 pop-os kernel: scsi host7: usb-storage 6-2:1.0
Jan 21 17:17:26 pop-os mtp-probe[10452]: checking bus 6, device 2: "/sys/devices/pci0000:00/0000:00:08.1/0000:0b:00.4/usb6/6-2"
Jan 21 17:17:26 pop-os mtp-probe[10452]: bus: 6, device: 2 was not an MTP device
Jan 21 17:17:26 pop-os mtp-probe[10467]: checking bus 6, device 2: "/sys/devices/pci0000:00/0000:00:08.1/0000:0b:00.4/usb6/6-2"
Jan 21 17:17:26 pop-os mtp-probe[10467]: bus: 6, device: 2 was not an MTP device
Jan 21 17:17:29 pop-os kernel: scsi 7:0:0:0: Direct-Access     Samsung  Flash Drive FIT  1100 PQ: 0 ANSI: 6
Jan 21 17:17:29 pop-os kernel: sd 7:0:0:0: Attached scsi generic sg1 type 0
Jan 21 17:17:29 pop-os kernel: sd 7:0:0:0: [sdb] 501253132 512-byte logical blocks: (257 GB/239 GiB)
Jan 21 17:17:29 pop-os kernel: sd 7:0:0:0: [sdb] Write Protect is off
Jan 21 17:17:29 pop-os kernel: sd 7:0:0:0: [sdb] Mode Sense: 43 00 00 00
Jan 21 17:17:29 pop-os kernel: sd 7:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesnt support DPO or FUA
Jan 21 17:17:29 pop-os kernel:  sdb: sdb1
Jan 21 17:17:29 pop-os kernel: sd 7:0:0:0: [sdb] Attached SCSI removable disk
Jan 21 17:17:41 pop-os systemd[1]: Starting Cryptography Setup for fit...
Jan 21 17:17:41 pop-os systemd-cryptsetup[10585]: Volume fit already active.
Jan 21 17:17:41 pop-os systemd[1]: Finished Cryptography Setup for fit.
Jan 21 17:17:41 pop-os systemd[1]: Reached target Block Device Preparation for /dev/mapper/fit.
Jan 21 17:17:41 pop-os gnome-shell[3744]: Unable to mount volume 257 GB Encrypted: Gio.IOErrorEnum: Operation was cancelled
Jan 21 17:17:41 pop-os udisksd[1382]: Unlocked device /dev/sdb1 as /dev/dm-4

Note that the second to last line "Operation was cancelled" is happening just before udisksd reports "Unlock device..."

I've tried this with two different devices, one with whole disk block encryption and the other with an encrypted partition, it makes no difference.

Sometimes moments later...

andy@pop-os:mnt$ mount /mnt/backup
mount: /mnt/backup: /dev/mapper/fit already mounted on /mnt/backup.

... but usually this just mounts the device as expected.

Other Issues

• If I do not set x-systemd.device-timeout to something much less than the default 90s then my system will slowly become unresponsive and then crash. I thought systemd might be timing out waiting for me to type in my passphrase but longer timeouts didn't seem to help
• I copied the configuration from an old Ubuntu 22.04 system to Pop!_OS 22.04, I would sometimes see the question marks while listing the directory in the unmounted state but the auto-mount worked perfectly. On the old system I didn't set the immutable attribute on the mountpoint.

Related

• mount-error-when-automounting-a-luks-encrypted-usb-flashdrive - different errors and method of mounting
• Automounting USB sticks on Debian - uses a slightly different method with nofail instead of noauto but unfortunately doesn't answer my question
• I can get more information with systemctl show mnt-backup.mount but this hasn't helped me
• The man page for fstab says what to do for external devices but these settings don't auto mount the device and do cause the boot process to hang

---

So, why does accessing the mountpoint try to access the device after it has been unmounted resulting in ?????????, and why does it not automount correctly when the device is inserted? Between the old fstab and new systemd I can't seem to find the magic variables.

Would also be useful to know why the system crashes if I don't set the device timeout?

---

Update August 2023

mount-vs-automount-systemd-units... explains the ????? as the .automount unit attempts to open the mountpoint on access.

I have still been unable to achieve the behaviour I want though, and wonder if it because without configuring anything udev is used and then the only problem is being able to customise the directory used as a mountpoint. As soon as an fstab entry is made, /run/systemd/generator systemd units are made and the behaviour changes.

There also seems to be a problem on my new system with plymouth as this spits out errors while systemd uses 100% CPU and hangs the system. This doesn't happen on my old laptop.

Answer 1

Just a quick idea...

Maybe use udev to launch script/app which make dir for your mount point and then do all cryptsetup stuff. This script may use key from usb drive.

Try do everything without using fstab.

(sorry for anwser, but I can't comment your question).

Answer 2

I have just achieved this, without udev or fstab. Machine is Arch Linux, LTS kernel, SwayWM, FWIW.

• First create a keyfile on your system, to store the password to your external device. This all comes from mighty Arch Wiki. Do these as root.

  dd bs=512 count=4 if=/dev/urandom of=/my_keyfile.bin
  chmod 000 /my_keyfile.bin
  cryptsetup luksAddKey /dev/sdb1 /my_keyfile.bin
  

• In your /etc/crypttab put:

  blah  UUID=blah-blah-blah-blah  /my_keyfile.bin  noauto
  

• Now, whether to mount it or not, leave it to your file manager. 

I'm using pcmanfm-qt, which has a daemon mode that you could autostart at boot (it seems Thunar also has this). So in your file manager's settings, choose whether to mount removable drives automatically or not.

Whether my USB SSD drive is present at boot or inserted later, it gracefully mounts for me without password.

Answer 3

I've come up with a solution that just about does what I wanted.

I recommend trying this out on spare devices so that your data is not at risk

The first thing to understand is how systemd-automount works. It does not open and mount the device when it is plugged in, but on access. Understanding this makes the setup much more comprehensible.

The second thing to note, is that the documentation (and even some utility programs) has not been updated for the systemd world.

With these two things in mind, you should know that systemd scans both /etc/crypttab and /etc/fstab and creates mount, automount, and service units as required. These can be found in /run/systemd/generator/

Most of the following commands require sudo access.

Preparation

Auto-mounting works best when you use a keyfile to unlock your LUKS container.

---

While not directly related to this question, there is no need for a LUKS encrypted usb drive to contain a partition. In fact, the recommendation is to create a raw encrypted device (citation needed)

---

You can see the current keyslots, and also a good idea to backup the header.

cryptsetup luksDump /dev/sdX
cryptsetup luksHeaderBackup --header-backup-file /root/<name>-header-backup /dev/sdX

Then create and add a new keyfile

dd if=/dev/urandom of=/root/<name>-keyfile bs=512 count=1
chmod 0400 /etc/cryptsetup-keys.d/crypt-backup.key
cryptsetup luksAddKey /dev/sdX /etc/cryptsetup-keys.d/crypt-backup.key

I have a passphrase and keyfile so I can automount my devices at home, but still access them on other machines which I wouldn't want to share the keyfile with.

The next thing I found really useful was to add a label to the luks header (and optionally the filesystem). This is possible with luks v2 and the label is visible to systemd before the device is opened (decrypted).

I use a blue/green backup strategy, and by labelling both external drives with the same label, I can get them to mount in the same place. Because a mount unit is created for each entry in fstab, the entries need to be unique. The LUKS header label is the only place I found I could do this.

# cryptsetup config /dev/sdX --label <label>
cryptsetup config /dev/sda --label backup
optional: also label the containing filesytem (must be open)
e2label /dev/mapper/<label> <fs-label>
e2label /dev/mapper/backup fit

Note that it is /dev/sdX, not /dev/sdX1, when not using partitions.

The Setup

While it makes sense to list /etc/crypttab first, and then /etc/fstab (because that's the order you'd use them), systemd works backwards. When you try to access a mountpoint it looks in fstab and finds the associated file-system. When it sees that it is /dev/mapper/something it then looks in crypttab.

While you could use something other than LABEL to identify your drive in crypttab, I couldn't find any other way for automounting to work than to use /dev/mapper/ in fstab.

# /etc/crypttab
backup LABEL=backup                              /etc/cryptsetup-keys.d/crypt-backup.key luks,nofail

This starts with the name you want to use for your crypt drive, "backup" in this example. Then you have to identify the device. Avoid using /dev/sdX type identifiers as these can change depending on what else you may or may not have plugged into your system. UUID is a good way to uniquely identify a device, but as I stated earlier, I want more than one device to be mounted the same way, so I use LABEL. Then, specify the location of the keyfile, or none is you want to use a passphrase. Finally, the options luks to identify it as a luks device, and nofail so that the device will be wanted but not required at boot.

# /etc/fstab
/dev/mapper/backup /mnt/backup ext4   noatime,users,nofail,x-systemd.device-timeout=100ms,x-systemd.automount,x-systemd.idle-timeout=20min  0  0

I've already talked about the need to use /dev/mapper/backup, then there is the mountpoing /mnt/backup which will need to exist, and the file system type. Now the options...

First of all, noauto, and auto don't do anyting in combination with x-systemd.automount. I said some documentation was outdated, you won't find nofail if you man fstab but that's what you want to use. Again, it stops your boot failing if the device isn't present. On my system there is a 90s timeout if devices are required but not found, however you could commpletely screw your normal boot process and need a recovery disk if not careful (possibly).

users allows the device to be mounted and umounted by any user. Normally, I'd set user so when I mount a device, I can also umount without sudo, but as systemd is doing the mounting we use users. Consider the implications of this for your own use-case.

Now for the real crunch x-systemd.device-timeout=100ms. When I go to the parent directory of my mountpoint and do ls -l I was getting loads of ????????. This is systemd saying that there should be something there but it can't find it. This way of working prevents writing anything to the mountpoint when the device is absent. However, you don't want to wait all day to list /mnt if several devices aren't plugged in (as would be normal). The x-systemd.idle-timeout=20min is optional.

Results

When you do a lsblk, or the first time you try to access the device, it may appear to be missing. But it will then automount and after a second or two, the device is ready to use.

lsblk
# NAME            TYPE  FSTYPE      FSVER      SIZE LABEL     MOUNTPOINTS
# sda             disk  crypto_LUKS 2        117.2G backup
ls -l /mnt/backup
ls: cannot access '/mnt/backup': No such device
ls -l /mnt/backup
total 24
drwxrwxr-x 3 andy andy  4096 Sep  5  2023 desktop
drwxrwxr-x 3 andy andy  4096 Sep  5  2023 desktop-backup
drwx------ 2 root root 16384 Aug  4  2023 lost+found
lsblk
NAME            TYPE  FSTYPE      FSVER      SIZE LABEL     MOUNTPOINTS
sda             disk  crypto_LUKS 2        117.2G backup
└─backup        crypt ext4        1.0      117.2G fit       /mnt/backup
ls /run/systemd/generator
mnt-backup.automount
mnt-backup.mount
systemd-cryptsetup@backup.service

You might need to do a systemctl daemon-reload and/or restart your system for the units to be generated.

Conclusion

This took me a while to figure and I hope my understanding is correct. I'm sure there is more to learn and if there are any corrections required please edit/comment.