Unable to `mkdir` in /mnt/ and cannot `chmod`

Question

I'm following this guide on my Ubuntu Azure hosted VM and need to make a directory in /mnt/. For this question I'll just change the directory I want to create to "test". I keep getting the following error message when running sudo mkdir /mnt/test/:

mkdir: cannot create directory ‘/mnt/test/’: Operation not permitted

I get a slightly different error message when I try to cd /mnt/.

-bash: cd: /mnt/: Permission denied

My next step was look into the permissions of this directory so I ran sudo stat /mnt/ and got:

File: /mnt/
Size: 4096          Blocks: 8          IO Block: 4096   directory
Device: 801h/2049d  Inode: 2           Links: 6
Access: (0444/dr--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-02-17 14:19:48.658231668 +0000
Modify: 2021-02-08 12:34:19.906967600 +0000
Change: 2021-02-09 09:40:12.287101300 +0000
Birth: -

I saw that I don't have write permissions so ran sudo chmod 664 /mnt/ but received the error message below:

chmod: changing permissions of '/mnt/': Operation not permitted

I'm still new to Unix so unsure how to proceed. Would appreciate any further troubleshooting steps or solutions.

Responses to comments:

df /mnt output:

Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/sda1       32894736 49192  31151544   1% /mnt

grep | '/mnt' output with some redacted information:

/dev/sda1 on /mnt type ext4 (rw,relatime,x-systemd.requires=cloud-init.service)
/dev/sda1 on /mnt/sdc1 type ext4 (rw,relatime)
/dev/sdc on /mnt/sdc type ext4 (rw,relatime)
//<azure storage account name>.file.core.windows.net/sih-workspace on /mnt/sih-workspace type cifs (rw,relatime,vers=3.0,cache=strict,username=**********,uid=0,noforceuid,gid=0,noforcegid,addr=<azure data centre IP>,file_mode=0777,dir_mode=0777,soft,persistenthandles,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,bsize=1048576,echo_interval=60,actimeo=1)

Please let me know if you need more information, I can run more commands if needed. — SocksSocksSocks, Feb 18 '21 at 12:27
Can you show us the output of df /mnt and mount | grep '/mnt' please? Is it a remote drive perhaps? What is the system you are using? Is it a virtual machine? — terdon, Feb 18 '21 at 13:10
Aside from chmod 644 being wrong for a directory (chmod 755 or even chmod u=rwx,go=rx would be better), I think that you've probably got something already mounted on /mnt. Please run the following command and add the result to your question, df /mnt — Chris Davies, Feb 18 '21 at 13:10
@terdon please see the output of those commands in my question. It's a file share contained an Azure storage account. The machine is running Ubuntu 18.04.4 and it's virtual. — SocksSocksSocks, Feb 18 '21 at 14:06
@roaima I tried your alternative chmods and both produced the same error message. Please see the output of df /mnt in my question. — SocksSocksSocks, Feb 18 '21 at 14:08
You seem to have several mounts on subdirectories of /mnt, which have now been hidden by another mount on /mnt itself. You need to decide which ones you want and which you don't expect. My suggestion would be that the first action should be to umount /mnt so that the others are exposed once more. But I am not familiar with the Azure filesystem utility you're using so this isn't a full answer — Chris Davies, Feb 18 '21 at 14:48
@roaima will that unmount everything? So from there it's like a blank slate and I can mount what I need again — SocksSocksSocks, Feb 18 '21 at 15:48
No. It will unmount the filesystem on /mnt, exposing the other mountpoints once more. — Chris Davies, Feb 18 '21 at 16:05
Facing the same issue on Azure instances!! what is the actual fix!? — Ani, Sep 29 '21 at 11:20

n0099 · Answer 1 · 2024-01-14T02:26:10.807

I encountered the same situation due to @chris-davies 's comment:

You seem to have several mounts on subdirectories of /mnt, which have now been hidden by another mount on /mnt itself. You need to decide which ones you want and which you don't expect. My suggestion would be that the first action should be to umount /mnt so that the others are exposed once more. But I am not familiar with the Azure filesystem utility you're using so this isn't a full answer

$ cat /etc/rc.local
mkdir /mnt/tmp
mount --bind /mnt/tmp /tmp
chmod 1777 /tmp
# shortly after booted
$ l /mnt
total 37K
drwxr-xr-x  5 root  root  4.0K Jan 14 08:46 .
drwxr-xr-x 19 root  root    26 Jan 14 01:22 ..
drwx------  2 root  root   16K Jan 14 08:46 lost+found
drwxrwxrwt  5 root  root  4.0K Jan 14 08:46 tmp
# just after systemd unit `ephemeral-disk-warning.service` started and failed
$ l /mnt
ls: cannot access '/mnt/.': Permission denied
ls: cannot access '/mnt/..': Permission denied
ls: cannot access '/mnt/lost+found': Permission denied
ls: cannot access '/mnt/tmp': Permission denied
total 0
d????????? ? ? ? ?            ? .
d????????? ? ? ? ?            ? ..
d????????? ? ? ? ?            ? lost+found
d????????? ? ? ? ?            ? tmp

Investigations:

$ journalctl --no-pager --boot -g ephemeral
Jan 14 08:46:48 azure systemd[1]: Starting Write warning to Azure ephemeral disk...
Jan 14 08:46:48 azure ephemeral-disk-warning[3758]: /usr/sbin/ephemeral-disk-warning: 7: cannot create /mnt
Jan 14 08:46:48 azure audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=ephemeral-disk-warning comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 14 08:46:48 azure root[3763]: Added ephemeral disk warning to /mnt
                                  /tmp/DATALOSS_WARNING_README.txt
Jan 14 08:46:48 azure systemd[1]: Finished Write warning to Azure ephemeral disk.
$ journalctl --no-pager --boot -u ephemeral-disk-warning
Jan 14 08:46:48 azure systemd[1]: Starting Write warning to Azure ephemeral disk...
Jan 14 08:46:48 azure ephemeral-disk-warning[3758]: /usr/sbin/ephemeral-disk-warning: 7: cannot create /mnt
Jan 14 08:46:48 azure ephemeral-disk-warning[3758]: /tmp/DATALOSS_WARNING_README.txt: Directory nonexistent
Jan 14 08:46:48 azure systemd[1]: Finished Write warning to Azure ephemeral disk.
Jan 14 08:46:48 azure ephemeral-disk-warning[3761]: chmod: cannot access '/tmp/DATALOSS_WARNING_README.txt': No such file or directory
Jan 14 08:46:48 azure ephemeral-disk-warning[3762]: chattr: No such file or directory while trying to stat /tmp/DATALOSS_WARNING_README.txt

$ systemctl cat ephemeral-disk-warning.service
[Unit]
Description=Write warning to Azure ephemeral disk
After=cloud-config.service
ConditionVirtualization=microsoft
ConditionPathIsMountPoint=/mnt
ConditionPathExists=/dev/disk/azure/resource-part1
[Service]
Type=oneshot
ExecStart=/usr/sbin/ephemeral-disk-warning
RemainAfterExit=yes
StandardOutput=journal+console
[Install]
WantedBy=multi-user.target

$ cat /usr/sbin/ephemeral-disk-warning
#!/bin/sh
dev_resource=$(readlink -f /dev/disk/azure/resource-part1)
dev_resource_mp=$(awk '$1==R {print$2}' "R=${dev_resource}" /proc/mounts)
warn_file="${dev_resource_mp}/DATALOSS_WARNING_README.txt"

if [ ! -f "${warn_file}" ]; then
    cat > ${warn_file} <<EOM
WARNING: THIS IS A TEMPORARY DISK.

Any data stored on this drive is SUBJECT TO LOSS and THERE IS NO WAY TO
RECOVER IT.

Please do not use this disk for storing any personal or application data.

For additional details to please refer to the MSDN documentation at:
http://msdn.microsoft.com/en-us/library/windowsazure/jj672979.aspx

To remove this warning run:
    sudo chattr -i $warn_file
    sudo rm $warn_file

This warning is written each boot; to disable it:
    echo "manual" | sudo tee /etc/init/ephemeral-disk-warning.override
    sudo systemctl disable ephemeral-disk-warning.service

EOM

    chmod 0444 ${warn_file}
    chattr +i ${warn_file}
    logger "Added ephemeral disk warning to ${warn_file}"
fi

$ source <(head -n 4 /usr/sbin/ephemeral-disk-warning)
$ echo $warn_file
/mnt
/tmp/DATALOSS_WARNING_README.txt
$ echo chattr +i ${warn_file}
chattr +i /mnt
/tmp/DATALOSS_WARNING_README.txt

So it's just yet another newline pitfall that leads to marking whole /mnt as immutable, finally I convert the old-style /etc/rc.local autorun script into a systemd unit:

# https://unix.stackexchange.com/questions/471824/what-is-the-correct-substitute-for-rc-local-in-systemd-instead-of-re-creating-rc
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/path/to/the/script.sh
StandardOutput=journal+console
StandardOutput=journal+console
[Unit]
https://unix.stackexchange.com/questions/635165/unable-to-mkdir-in-mnt-and-cannot-chmod/766859#766859
After=ephemeral-disk-warning.service
After=mnt.mount
Requires=mnt.mount
RequiredBy=anything-requiring-mnt.service
Before=anything-requiring-mnt.service
[Install]
WantedBy=multi-user.target

You really should file an issue with Microsoft to get their shitty script fixed, if you're a customer. That said, I don't understand what the final systemd unit is for. Which script is it executing and what is that script supposed to be doing? — muru, Jan 14 '24 at 02:17
Requesting support tickets that not related to billing issues requires a subscription, if they really wanna fix this: quick n'dirty: Insert | head -n 1 after /proc/mounts or more comprehension: Using for to iterate each newline-separated(but still having possible to get \n in path) path in $dev_resource_mp then wrapping $warn_file and that following if block into it. — n0099, Jan 14 '24 at 02:33
@muru The systemd unit I created is just a wrapper around /etc/rc.local that moved to /path/to/the/script.sh to make sure it only gets started AFTER ephemeral-disk-warning.service created the /mnt/DATALOSS_WARNING_README.txt without silly chattr +i /mnt then exited and other units that using paths under /mnt also wait till the unit exited. — n0099, Jan 14 '24 at 02:39

score -1 · Answer 2 · answered Aug 24 '23 at 19:56

If you have a mounted filesystem on /mnt you can't create anything under it. You can test this by running "$ df -h /mnt" I can assume that you will see an auto map such as this:

Filesystem Size Used Avail Use% Mounted on

/etc/auto.toybox 0 0 0 - /mnt

If that is so, then the automounter has claimed /mnt as its mount point for the auto mapper and nothing you ever try to create under /mnt will work. Thats why the permission is denied.

If you see an automount, like above, you've probably done an incorrect direct mapping for the automounter. Look at the autofs and ypbind man and systemctl outputs. Use the ypcat -k command to figure out what the auto.master map is doing.

Use ypcat -k auto.master or ypcat -k auto.home, etc... so you understand the mappings.
Remember that auto.master employs maps such as auto.home which may be in your /etc/ypsrc/ directory if using a redhat variant and ypbind. See your maps using 'automount -m' or 'sudo automount -m ' Any changes you make to the maps on the NIS server will need you to run 'make' from /var/yp/

Using the /etc/ypsrc/auto.home will allow the automapper to mount the directory to the client:/home directory so that if you 'really' need to mkdir in /mnt then that should have moved the mount point to /home and now allow you to mkdir in /mnt without the dreaded 'permission denied' error... but you really should not use /mnt for making random directories - imho. Summarizing, you cant mkdir on a used mountpoint and its probably an automount / autofs issue OR the wrong owner/group is assigned to /mnt < which is pretty easy to notice and fix.

Unable to `mkdir` in /mnt/ and cannot `chmod`

2 Answers2

https://unix.stackexchange.com/questions/635165/unable-to-mkdir-in-mnt-and-cannot-chmod/766859#766859