1

I am really struggling to get to the bottom of this one.

For some reason cron cannot see a file system I mounted manually. This is a USB drive formatted to ext4 mounted to /backup. For completeness I mounted it while logged into SSH, not directly at a terminal.

If I compare typing mount | sort at the commandline (over ssh) with the same command over cron or atd the cron version will miss the lines:

tmpfs on /run/sshd type tmpfs (rw,nosuid,nodev,mode=755)
/dev/sdb1 on /backup type ext4 (rw,relatime,data=ordered)

I've confirmed that neither cron nor sshd are chrooted using the accepted answer to another question.

If neither are chrooted, then what else can cause two different processes to see different mounts?

This is really causing a problem because my backups keep writing to the 30GB SD card in my r-pi instead of the 2TB USB hard drive.

Edit for future reference. This bug in Systemd v236 looks like the cause. https://github.com/systemd/systemd/issues/7761

If I type 

mount | sort

I get 

/dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
/dev/mmcblk0p2 on / type ext4 (rw,noatime,data=ordered)
/dev/sda1 on /home type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /mnt/mercury_data type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /root type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /var type ext4 (rw,relatime,data=ordered)
/dev/sdb1 on /backup type ext4 (rw,relatime,data=ordered)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devtmpfs on /dev type devtmpfs (rw,relatime,size=470180k,nr_inodes=117545,mode=755)
mqueue on /dev/mqueue type mqueue (rw,relatime)
proc on /proc type proc (rw,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/sshd type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)

But when I run this through cron or atd I get:

/dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
/dev/mmcblk0p2 on / type ext4 (rw,noatime,data=ordered)
/dev/sda1 on /home type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /mnt/mercury_data type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /root type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /var type ext4 (rw,relatime,data=ordered)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devtmpfs on /dev type devtmpfs (rw,relatime,size=470180k,nr_inodes=117545,mode=755)
mqueue on /dev/mqueue type mqueue (rw,relatime)
proc on /proc type proc (rw,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
Philip Couling
  • 18,733

1 Answers1

3

This sounds like the daemons are running in different mount namespaces, so changes you make in your SSH session aren’t visible in cron or at jobs. Look at mountinfo and ns/mnt inside the various deamons’ /proc/${pid} directories to check which namespaces they’re using and what they can inherit.

Stephen Kitt
  • 434,908
  • yes, on the ssh command line ls -lh /proc/1/ns/mnt /proc/self/ns/mnt shows different numbers. This shows my SSH session is in a different namespace to the main system. Further digging, my cron daemon is in the main (same as init) namespace so the problem is the way sshd is being loaded. Thanks for your help! – Philip Couling Feb 12 '18 at 15:56
  • Okay. Lots more digging. Looks like a known bug in systemd for this version. https://github.com/systemd/systemd/issues/7761 – Philip Couling Feb 12 '18 at 16:41
  • Nice digging! It does look like that’s the issue... – Stephen Kitt Feb 12 '18 at 16:45