unable to remount filesystem back to read-only after package upgrade

Question

On my Debian system, I have my / on separate partition and mounted read-only. Only /home/, /var/ and /tmp/ are writable. I have also created a Pre-Invoke and Post-Invoke apt hook, so that apt can automatically remount the system for write, when packages are installed or upgraded, and remount it back to read-only when finished:

DPkg::Pre-Invoke  {"mount -o remount,rw / ;};
DPkg::Post-Invoke {"mount -o remount    / ;};

This whole setup works well with one exception. Sometimes during the installation/upgrade process some services need to be restarted or new files opened during the short window when my / is mounted read-write, these files are opened with write permissions. After the installation/upgrade is complete, my Post-Invoke hook returns error because it cannot remount / back to read-only.

Is there some way to solve this problem? This is very annoying, because in this situation I usually must restart the server, which is impractical.

EDIT

Below is a log of my latest package upgrade, which resulted in the described error:

root@alpha# apt-get upgrade 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be upgraded:
  base-files curl libc-bin libc6 libcurl3 libcurl3-gnutls libmysqlclient18 libssl1.0.0 locales multiarch-support mysql-client mysql-client-5.5 mysql-common
  nscd openssl tzdata wget whois
18 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 18.7 MB of archives.
After this operation, 264 kB of additional disk space will be used.
Do you want to continue [Y/n]? 
Fetched 18.7 MB in 0s (33.2 MB/s) 
Preconfiguring packages ...
(Reading database ... 20532 files and directories currently installed.)
Preparing to replace base-files 7.1wheezy3 (using .../base-files_7.1wheezy4_amd64.deb) ...
Unpacking replacement base-files ...
Processing triggers for man-db ...
Processing triggers for install-info ...
Setting up base-files (7.1wheezy4) ...
Installing new version of config file /etc/debian_version ...
(Reading database ... 20532 files and directories currently installed.)
Preparing to replace libc-bin 2.13-38 (using .../libc-bin_2.13-38+deb7u1_amd64.deb) ...
Unpacking replacement libc-bin ...
Processing triggers for man-db ...
Setting up libc-bin (2.13-38+deb7u1) ...
(Reading database ... 20532 files and directories currently installed.)
Preparing to replace libc6:amd64 2.13-38 (using .../libc6_2.13-38+deb7u1_amd64.deb) ...
Unpacking replacement libc6:amd64 ...
Setting up libc6:amd64 (2.13-38+deb7u1) ...
(Reading database ... 20532 files and directories currently installed.)
Preparing to replace libssl1.0.0:amd64 1.0.1e-2+deb7u1 (using .../libssl1.0.0_1.0.1e-2+deb7u4_amd64.deb) ...
Unpacking replacement libssl1.0.0:amd64 ...
Preparing to replace curl 7.26.0-1+wheezy7 (using .../curl_7.26.0-1+wheezy8_amd64.deb) ...
Unpacking replacement curl ...
Preparing to replace libcurl3:amd64 7.26.0-1+wheezy7 (using .../libcurl3_7.26.0-1+wheezy8_amd64.deb) ...
Unpacking replacement libcurl3:amd64 ...
Preparing to replace libcurl3-gnutls:amd64 7.26.0-1+wheezy7 (using .../libcurl3-gnutls_7.26.0-1+wheezy8_amd64.deb) ...
Unpacking replacement libcurl3-gnutls:amd64 ...
Preparing to replace mysql-common 5.5.33+dfsg-0+wheezy1 (using .../mysql-common_5.5.35+dfsg-0+wheezy1_all.deb) ...
Unpacking replacement mysql-common ...
Preparing to replace libmysqlclient18:amd64 5.5.33+dfsg-0+wheezy1 (using .../libmysqlclient18_5.5.35+dfsg-0+wheezy1_amd64.deb) ...
Unpacking replacement libmysqlclient18:amd64 ...
Preparing to replace multiarch-support 2.13-38 (using .../multiarch-support_2.13-38+deb7u1_amd64.deb) ...
Unpacking replacement multiarch-support ...
Processing triggers for man-db ...
Setting up multiarch-support (2.13-38+deb7u1) ...
(Reading database ... 20532 files and directories currently installed.)
Preparing to replace tzdata 2013h-0wheezy1 (using .../tzdata_2013i-0wheezy1_all.deb) ...
Unpacking replacement tzdata ...
Setting up tzdata (2013i-0wheezy1) ...

Current default time zone: 'Europe/London'
Local time is now:      Sat Feb 15 11:35:41 CET 2014.
Universal Time is now:  Sat Feb 15 11:35:41 UTC 2014.
Run 'dpkg-reconfigure tzdata' if you wish to change it.

(Reading database ... 20511 files and directories currently installed.)
Preparing to replace wget 1.13.4-3 (using .../wget_1.13.4-3+deb7u1_amd64.deb) ...
Unpacking replacement wget ...
Preparing to replace locales 2.13-38 (using .../locales_2.13-38+deb7u1_all.deb) ...
Unpacking replacement locales ...
Preparing to replace whois 5.0.23 (using .../whois_5.1.1~deb7u1_amd64.deb) ...
Unpacking replacement whois ...
Preparing to replace mysql-client 5.5.33+dfsg-0+wheezy1 (using .../mysql-client_5.5.35+dfsg-0+wheezy1_all.deb) ...
Unpacking replacement mysql-client ...
Preparing to replace mysql-client-5.5 5.5.33+dfsg-0+wheezy1 (using .../mysql-client-5.5_5.5.35+dfsg-0+wheezy1_amd64.deb) ...
Unpacking replacement mysql-client-5.5 ...
Preparing to replace nscd 2.13-38 (using .../nscd_2.13-38+deb7u1_amd64.deb) ...
[ ok ] Stopping Name Service Cache Daemon: nscd.
Unpacking replacement nscd ...
Preparing to replace openssl 1.0.1e-2+deb7u1 (using .../openssl_1.0.1e-2+deb7u4_amd64.deb) ...
Unpacking replacement openssl ...
Processing triggers for install-info ...
Processing triggers for man-db ...
Setting up libssl1.0.0:amd64 (1.0.1e-2+deb7u4) ...
Setting up libcurl3:amd64 (7.26.0-1+wheezy8) ...
Setting up curl (7.26.0-1+wheezy8) ...
Setting up libcurl3-gnutls:amd64 (7.26.0-1+wheezy8) ...
Setting up mysql-common (5.5.35+dfsg-0+wheezy1) ...
Setting up libmysqlclient18:amd64 (5.5.35+dfsg-0+wheezy1) ...
Setting up wget (1.13.4-3+deb7u1) ...
Setting up locales (2.13-38+deb7u1) ...
Generating locales (this might take a while)...
  en_DK.UTF-8... done
  en_US.UTF-8... done
Generation complete.
Setting up whois (5.1.1~deb7u1) ...
Setting up mysql-client-5.5 (5.5.35+dfsg-0+wheezy1) ...
Setting up mysql-client (5.5.35+dfsg-0+wheezy1) ...
Setting up nscd (2.13-38+deb7u1) ...
[ ok ] Starting Name Service Cache Daemon: nscd.
Setting up openssl (1.0.1e-2+deb7u4) ...
mount: / is busy

The last line (mount: / is busy) is the error returned by apt, when trying to remount / back to read-only.

UPDATE:

the command suggested by Graeme does not show any files:

# lsof / | awk 'NR==1 || $4~/[0-9][uw]/'
COMMAND     PID       USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME

Are you looking for a way to prevent the files from being opened read-write, or from having to reboot the server while being able to remount ro, or finding and changing the blocking packages? Or would either be acceptable as a solution? — Anthon, Feb 15 '14 at 11:08
Ideally, I would like to prevent the files from being opened rw in the first place. But any solution that will enable me to remount to ro without a restart is also good. — Martin Vegter, Feb 15 '14 at 11:18
@martin do you already have an overview of the offending services? I like the setup you presented and will try that in a VM, but it would be nice to know you are not running something non-default on your system that would make my experiments only partly relevant. — Anthon, Feb 15 '14 at 11:22
@Anthon I don't know which are the offending services. But see my EDIT above for clarification. My server is also a VM. It is a minimal installation with only few services running. — Martin Vegter, Feb 15 '14 at 12:01
Did you see my comment below? Have you tried running the lsof command as part of the Post-Invoke hook? — Graeme, Feb 21 '14 at 17:22

score 2 · Answer 1 · edited Feb 22 '14 at 03:14

My guess is that it's not just services, it's the fact that you have other filesystems, like /home and /var, mounted inside the root filesystem. That aside, the best solution I could dig up is outlined here:

https://sites.google.com/site/linuxpendrive/rorootfs

Look for the section entitled How do I install/uninstall packages on a read only file system? In a nutshell it involves remounting the target filesystem and then chrooting into the new mount, before using the package manager.

The suggestion, outlined in one of the other answers, makes assumptions as to what happens when the root filesystem is remounted rw for package updates, so this solution may not actually work Debian, if Debian exhibits different behaviour than what is assumed. But hey, it's worth a shot, I think...

score 1 · Accepted Answer · edited Apr 13 '17 at 12:36

To get closer to a definitive answer for this, we need to see which files are causing the mount: / is busy error. You could do this with:

lsof / | awk 'NR==1 || $4~/[0-9][uw]/'

See my answer to the OP's other question - lsof: show files open as read-write - for the caveats to this. It may be that you need to put this into a separate script and the put the script into the apt hook in order to see something.

My suspicion is that files under /etc remain open once the services are started. Some programs/daemons update their configuration dynamically. NetworkManager and cupsd are two examples. Updates to cups which cause cupsd to scan for new printers (as opposed to a dpkg configuration script) may well be what is causing your issue. I recommend that you put /etc on a writeable file system, even if it is not the source of your problem.

Another possibility is that the filesystem buffer is still in the process of being flushed to disk when you try to do the remount. I'm not sure what the behaviour for mount is here, whether it is to block until IO is complete or to fail and report the disk as busy. The first seems more likely, but I see no sync calls in the output of strace (although possibly the mount system call does this). Anyway, it may work to do a sync before the remount if the lsof above doesn't show anything, eg:

DPkg::Post-Invoke { "sync; mount -o remount /"; };

this is strange. Your lsof command does not list any files — Martin Vegter, Feb 19 '14 at 23:02
Interesting, one other thing is that memory mapped files can be copy-on-write. I don't know how this affects mounting, plus I'm not sure lsof and even show this. Did you try running lsof with your hook? How about the sync stuff? — Graeme, Feb 19 '14 at 23:41

unable to remount filesystem back to read-only after package upgrade

EDIT

UPDATE:

2 Answers2

Linked