13

I want to replace /home with a symlink to my nfs-mounted home dirs.

Only root is logged in, /home is not a separate filesystem, lsof shows no locks, selinux is permissive. What am I missing?

I'm logged in directly as root via ssh:

[root@usil01-sql01 /]# uname -a
Linux usil01-sql01 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@usil01-sql01 /]# w
 15:30:33 up  1:41,  1 user,  load average: 0.00, 0.02, 0.22
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/2    10.50.11.114     15:13    1.00s  0.19s  0.01s w

[root@usil01-sql01 /]# lsof | grep /home

[root@usil01-sql01 /]# lsof +D /home

[root@usil01-sql01 /]# df -h /home
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        63G  4.1G   56G   7% /

[root@usil01-sql01 /]# mount | grep -w /
/dev/sda2 on / type ext4 (rw,relatime,seclabel,data=ordered)

[root@usil01-sql01 /]# ls -lFd /home
drwxr-xr-x. 3 root root 4096 Mar  7 13:36 /home/

[root@usil01-sql01 /]# getenforce
Permissive

[root@usil01-sql01 /]# mv /home /home-old
mv: cannot move "/home" to "/home-old": Device or resource busy

What else can I check?

More system info:

[root@usil01-sql01 /]# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 836.6G  0 disk 
|-sda1   8:1    0 768.6G  0 part /storage
|-sda2   8:2    0    64G  0 part /
`-sda3   8:3    0     4G  0 part [SWAP]
sr0     11:0    1  1024M  0 rom  

[root@usil01-sql01 /]# blkid
/dev/sda2: UUID="5ba6a429-4c65-4023-82b4-3673bfcf6a88" TYPE="ext4" 
/dev/sda3: UUID="b5eb680f-8789-43b2-9f7e-c52570b0eb73" TYPE="swap" 
/dev/sda1: UUID="cb22d57d-4a5b-4963-a990-890abe0c56dc" TYPE="ext4"
TheAmigo
  • 383
  • Try the lazy umount option as in umount -f -l /home – Valentin Bajrami Mar 07 '17 at 21:41
  • Try to look up a suspicious process that consumes to much resources while not being known to you with top or ps. Them try to watch what it's doing right now using strace. That situation is weird tmm. UPDATE: also double check lsof output with fuser one just in case. – ddnomad Mar 07 '17 at 21:48
  • 1
    @val0x00ff as you can see in the output above, /home is not a mount point. – TheAmigo Mar 07 '17 at 22:17
  • 1
    @ddnomad fuser /home also has no output. It's a freshly-installed system sitting idle, no busy processes. – TheAmigo Mar 07 '17 at 22:17
  • @TheAmigo I see. I presume you've already tried to simply reboot the host. You might also try to boot into a safe mode from GRUB menu and try it there. – ddnomad Mar 07 '17 at 22:24
  • @TheAmigo show some more details blkid, lsblk, lvs, lvdisplay etc. – Valentin Bajrami Mar 07 '17 at 22:33
  • @ddnomad Yes, I tried a reboot, no change. No, I've not yet tried alternate boot methods, getting console requires a trip to the data center. – TheAmigo Mar 08 '17 at 14:12
  • @val0x00ff Added the output ob blkid and lsblk, not very interesting. No logical volumes... very vanilla. – TheAmigo Mar 08 '17 at 14:14

3 Answers3

13

mv: cannot move "/home" to "/home-old": Device or resource busy

The only "use"[*] I can think of, which holds the name of a file from changing, is a mount point.

What else can I check?

I am not certain, but perhaps this could happen if the mount still exists in another mount namespace. Because it's not getting unmounts propagated from the root namespace, for some reason? Or looking at the result on my system, maybe systemd services with ProtectHome?

$ grep -h home /proc/*/task/*/mountinfo | sort -u
121 89 0:22 /systemd/inaccessible/dir /home ro,nosuid,nodev shared:142 master:24 - tmpfs tmpfs rw,seclabel,mode=755
275 243 253:2 / /home ro,relatime shared:218 master:33 - ext4 /dev/mapper/alan_dell_2016-home rw,seclabel,data=ordered
321 288 253:2 / /home rw,relatime shared:262 master:33 - ext4 /dev/mapper/alan_dell_2016-home rw,seclabel,data=ordered
84 64 253:2 / /home rw,relatime shared:33 - ext4 /dev/mapper/alan_dell_2016-home rw,seclabel,data=ordered
85 46 253:2 / /home rw,relatime master:33 - ext4 /dev/mapper/alan_dell_2016-home rw,seclabel,data=ordered

Note this issue - unable to rename /home despite it not showing as a mount point (in the current namespace) - should be fixed in Linux kernel version 3.18+.

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-3.18.y&id=8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe

how to find out namespace of a particular process?

lsns might be useful if you can install it. More possible commands:

List mount namespaces:

# readlink /proc/*/task/*/ns/mnt | sort -u

Identify root mount namespace:

# readlink /proc/1/ns/mnt

Find processes with a given mount namespace

# readlink /proc/*/task/*/ns/mnt | grep 4026531840

Inspect the namespace of a given process:

# cat /proc/1/task/1/mountinfo

[*] EBUSY The rename fails because oldpath or newpath is a directory that is in use by some process (perhaps as current working directory, or as root directory, or because it was open for reading) or is in use by the system (for example as mount point), while the system considers this an error. (Note that there is no require‐ ment to return EBUSY in such cases—there is nothing wrong with doing the rename anyway—but it is allowed to return EBUSY if the system cannot otherwise handle such situations.)

sourcejedi
  • 50,249
  • /home has never been a mount in any namespace. – TheAmigo Mar 08 '17 at 14:14
  • Does that mean you used commands along the lines I suggested? Because "never" is a very strong assertion, my commands wouldn't demonstrate that. I've added a command at the start which would be simpler (and hopefully functional :), and a second speculation on why this might happen based on the result on my system. – sourcejedi Mar 08 '17 at 15:00
  • I found another "broken" machine and your grep command indeed points the finger at NetworkManager. So my "stop NM, rename, restart NM" works, but your grep of mountinfo is what find the culprit. – TheAmigo Mar 08 '17 at 16:22
  • (because /lib/systemd/system/NetworkManager.service uses ProtectHome, at least on the systems we're looking at) – sourcejedi Mar 08 '17 at 16:39
8

It was NetworkManager.

Running systemctl stop mysqld httpd postfix ipmievd tuned atd rsyslog smartd crond irqbalance gssproxy polkit chronyd didn't help, but made the process table very small.

After systemctl stop NetworkManager, I was able to rename /home.

TheAmigo
  • 383
  • I needed to do only "service NetworkManager stop" and then was able to move /home then did a reboot to make sure all was well. – sdjuan Apr 09 '19 at 22:49
  • Thank you very much. This is not at all obvious and I had to read dozens of pages talking about /home as a mount point, which is clearly not in the OP question. I found stopping NetworkManager also helped in my case, running Redhat 7.6. – labradort Jul 08 '19 at 13:00
  • You just saved one of my peers an entire afternoon of grief, I think. I already hated NetworkManager, so this was oddly cheering. He's on CentOS 7.8 – Alex North-Keys Jun 05 '20 at 19:07
  • Thanks for the tip. For me, I just had to stop chronyd systemctl stop chronyd – IanB Jan 13 '21 at 23:29
0

You can boot on single user and make any changes on home directory.

  1. Rebooted my system and durring grub selection edit the line by pressing e
  2. In linux16 line removed rhgb & quite options and put in their placed init=/bin/bash.
  3. The pressed ctrl+x to start. This will promt you on bash console.
  4. Remount / with read/write options by issuing mount -o remount,rw /
  5. After that you can edit your /home directory rename it etc.
  6. After work is done relabel the selinux by issuing touch /.autorelabel
  7. Finally execute exec /sbin/init to start boot as normal.

@sourcejedi: thank you for your answer.

igiannak
  • 750
  • 1
    While that works for local machines, it can be a challenge for a physical machine in a remote data center unless you have an IPMI or some means of remote access to the console. – TheAmigo Jan 21 '21 at 19:13