0

Both systemd and Bluetooth processes on my ASUS Tinkerboard (Debian 10) are utilizing a deleted rfkill file. I suspect this might be the cause of the storage appearing as 100% full when checking with df -h.

How can I restore a valid rfkill file or stop the processes that are using this deleted file?

I cannot remove and reinstall rfkill, because I get this error (the disk is full with the deleted file being in use):

dpkg: unrecoverable fatal error, aborting:
 unable to fill /var/lib/dpkg/updates/tmp.i with padding: No space left on device

The alternative route to stop the processes works easily for Bluetooth, but systemd is such a core process (it has a pid number of 1) that I can't imagine how to do it:

lsof +L1
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
systemd   1 root   89u   CHR 10,242      0t0     0 1171 /dev/rfkill (deleted)

The disk is 16GB and shows as being full with df -h, but du -kx / | sort -rn | less shows only 3.5GB. Here are the first 50 lines apporximately:

3455876 /
2498820 /usr
1174128 /usr/lib
1054352 /usr/share
806276  /usr/lib/aarch64-linux-gnu
556544  /var
247288  /var/cache
238792  /var/cache/apt
225056  /usr/share/locale
198740  /home
198736  /home/linaro
190568  /usr/share/doc
187660  /usr/lib/aarch64-linux-gnu/dri
183944  /usr/bin
179352  /var/lib
177304  /var/cache/apt/archives
148316  /home/linaro/.cache
147356  /home/linaro/.cache/chromium
147352  /home/linaro/.cache/chromium/Default
146584  /home/linaro/.cache/chromium/Default/Code Cache
146564  /home/linaro/.cache/chromium/Default/Code Cache/js
122204  /var/lib/apt
122132  /var/lib/apt/lists
112628  /usr/share/fonts
104228  /var/log
102284  /lib
84692   /usr/share/fonts/truetype
77888   /usr/share/icons
74312   /usr/share/emacs
74264   /usr/share/emacs/26.1
73184   /usr/lib/python2.7
72984   /usr/lib/gcc
72980   /usr/lib/gcc/aarch64-linux-gnu
72976   /usr/lib/gcc/aarch64-linux-gnu/8
65860   /usr/lib/python3
65856   /usr/lib/python3/dist-packages
59900   /usr/share/emacs/26.1/lisp
56344   /lib/modules
56340   /lib/modules/4.19.193
56240   /lib/modules/4.19.193/kernel
56236   /lib/modules/4.19.193/kernel/drivers
54708   /lib/modules/4.19.193/kernel/drivers/net
54704   /lib/modules/4.19.193/kernel/drivers/net/wireless
52280   /usr/lib/python2.7/config-aarch64-linux-gnu
50100   /home/linaro/.config
49876   /home/linaro/.config/chromium
48744   /usr/include
48280   /usr/share/scratch
44276   /var/lib/dpkg
41632   /usr/share/icons/Adwaita
41396   /var/lib/dpkg/info
41316   /home/linaro/.config/chromium/Default
40312   /usr/lib/python3/dist-packages/scipy
39964   /lib/modules/4.19.193/kernel/drivers/net/wireless/rockchip_wlan
39952   /usr/lib/python3.7
37324   /usr/share/fonts/truetype/arphic
35040   /home/linaro/.config/chromium/Default/Service Worker
Dzseti
  • 121
  • https://unix.stackexchange.com/questions/5642/what-if-kill-9-does-not-work confirms that I cannot end the core process: kill -9 1 will not work. So what do I do? – Dzseti Jun 30 '23 at 07:25

2 Answers2

2

/dev/rfkill should be a character device, not a regular file. So it should not actually cause a disk to become full, unless it was first (erroneously) replaced with a regular file. Your lsof +L1 results also indicate that systemd is using the proper /dev/rfkill device node: TYPE is CHR and DEVICE is 10,242 just like with a real intact /dev/rfkill:

ls -l /dev/rfkill
crw-rw-r-- 1 root netdev 10, 242 Jun 14 20:59 /dev/rfkill

Note the cas the first character indicating this is not a regular file, but a character device node, and the major and minor device numbers (10 and 242 respectively) in place of a file size - because a device node is essentially just a directory entry, it does not have a meaningful size. The numbers tell the kernel which in-kernel device driver this device node is supposed to communicate with.

The list of device node numbers in Linux kernel documentation can tell us that /dev/rfkill is supposed to be a character device node with major 10 and minor 242.

You could recreate a deleted /dev/rfkill device node pretty simply with two commands as root:

mknod -m 664 /dev/rfkill c 10 242
chgrp netdev /dev/rfkill

(All the contents of /dev live in RAM are not on any actual disk: they are recreated on every boot.)

On the other hand, if the /usr/sbin/rfkill command has been deleted, you'll need dpkg to reinstall it. But if your disk is 100% full, you cannot do that without first making some free disk space.

Does your lsof +L1 indicate any other deleted-but-stil-in-use files (with a type REG = regular file)?

If you can easily re-download things from Debian repositories, running apt-get clean can be an easy way to quickly get some free space: it will clear the local cache of downloaded packages without affecting any installed software.

telcoM
  • 96,466
  • Thanks @telcoM This detailed explanation is very helpful at suggesting where to go next. Recreating the /dev/rfkill device node solves the deleted-file-in-use issue (there are no others), but if I reboot, then the /dev/rfkill problem returns. Your apt-get clean suggestion was useful: it changed the disk usage from 15G to 14G, but still showing 100%. Then at reboot even more deleted-file-in-use lines appeared including 6 deleted files relating to apache2 – Dzseti Jun 30 '23 at 08:45
  • The /dev/rfkill device node should be automatically created by udev on loading the rfkill kernel module: is that module currently loaded? It should be, if you have the Bluetooth subsystem active: the bluetooth kernel module depends on rfkill. Or is something forcing rfkill to unload? That would be highly unusual. – telcoM Jun 30 '23 at 08:49
  • lsmod only shows 2 modules as loaded: 8822ce and rtk_btusb. I wonder whether the apparent lack of space on the disk is causing this – Dzseti Jul 01 '23 at 06:50
  • If you are running a standard Debian kernel, having only 2 modules loaded would be very unusual. Perhaps the disk space issue is the root cause. Try du -kx / | sort -rn | less to list directories in order of decreasing size. Of course it will list / at the top as it includes everything else, but look into the deepest directory paths near the top of the listing to find the largest disk space consumers. If you can't find anything, please edit your original question and post the first 50 or so lines of the output to it. – telcoM Jul 01 '23 at 10:17
  • Original question edited as requested @telcoM – Dzseti Jul 02 '23 at 07:09
  • Further information (should I start a new question for this?): on trying a fsck at boot I get two error messages: systemd-sysctl[195]: Couldn't write 'force' to 'fsck/mode', ignoring: No such file or directory and systemd-sysctl[195]: Couldn't write 'yes' to 'fsck/repair', ignoring: No such file or directory. The check definitely doesn't run, because I can see the last cehck date from tune2fs -l /dev/mmcblk2p9 | grep 'checked\|Maximum\|Mount' which shows Last checked: Tue May 3 13:10:09 2022 – Dzseti Jul 03 '23 at 06:36
  • You should add those to your original question, or maybe start a new one. systemd-sysctl suggests it's reading configuration files from /etc/sysctl.d and trying to write those settings to the kernel via the virtual /proc/sys filesystem. But things like fsck/mode or fsck/repair do not make sense in that context, as fsck is an user-space application and the kernel should have nothing to do with its settings, so there's probably something in /etc/sysctl.d/ that doesn't actually belong there. – telcoM Jul 03 '23 at 07:36
1

After moving the root of the Tinkerboard 2S to a Micro SD rather than the built-in eMMC and setting up a mirror system, I noticed that there were almost 12GB of files that should have been on a mounted drive not on the eMMC itself

The mount had obviously failed sometime and the download software did not notice this and so saved files to the local storage rather than the remote mount

I only noticed this because I had moved the mount to the Micro SD card, which left only the files saved locally on the eMMC

Lessoned learned: I should have checked everything without mounting the external drives

For the future I have followed the answer here to prevent writing to a directory where a mount has failed

Dzseti
  • 121