What is the exact purpose of `mask` in file system ACL?

Question

When I apply default ACL in a directory I see default:mask or just mask in the following two scenario.

Scenario 1

-bash-4.2$ ls -ld test/
 drwxr-x---. 2 test test 4096 Oct 15 19:12 test/

-bash-4.2$ setfacl -d -m u:arif:rwx test/

-bash-4.2$ getfacl --omit-header test
 user::rwx
 group::r-x
 other::---
 default:user::rwx
 default:user:arif:rwx
 default:group::r-x
 default:mask::rwx
 default:other::---

Scenario 2

-bash-4.2$ ls -dl dir/
 drwxr-x---. 2 test test 4096 Oct 15 18:17 dir/

-bash-4.2$ getfacl dir
 # file: dir
 # owner: test
 # group: test
 user::rwx
 group::r-x
 other::---

-bash-4.2$ setfacl -m user:arif:rwx dir

-bash-4.2$ getfacl --omit-header dir
 user::rwx
 user:arif:rwx
 group::r-x
 mask::rwx
 other::---

So what is the purpose of mask here?

Answer 1

What

This 3-bit ACL system has its roots in TRUSIX. Other ACL systems, such as the NFS4-style ones in FreeBSD, MacOS, AIX, Illumos, and Solaris, work differently and this concept of a mask access control entry is not present.

The mask is, as the name says, a mask that is applied to mask out permissions granted by access control entries for users and groups. It is the maximum permission that may be granted by any acccess control entry, other than by a file owner or an "other" entry. Its 3 bits are anded with the 3 bits of these other entries.

So, for example, if a user is granted rw- by an access control entry, but the mask is r--, the user will only actually have r-- access. Conversely, if a user is only granted --x by an access control entry, a mask of rwx does not grant extra permissions and the user has just --x access.

The default mask on a parent directory is the mask setting that is applied to things that are created within it. It is a form of inheritance.

Why

It's a shame that IEEE 1003.1e never became a standard and was withdrawn in 1998. In practice, nineteen years on, it's a standard that a wide range of operating systems — from Linux through FreeBSD to Solaris (alongside the NFS4-style ACLs in the latter cases) — actually implement.

IEEE 1003.1e working draft #17 makes for interesting reading, and I recommend it. In appendix B § 23.3 the working group provides a detailed, eight page, rationale for the somewhat complex way that POSIX ACLs work with respect to the old S_IRWXG group permission flags. (It's worth noting that the TRUSIX people provided much the same analysis ten years earlier.) This covers the raison d'être for the mask, which I will only précis here.

• Traditional Unix applications expect to be able to deny all access to a file, named pipe, device, or directory with chmod(…,000). In the presence of ACLs, this only turns off all user and group permissions if there is a mask and the old S_IRWXG maps to it. Without this, setting the old file permissions to 000 wouldn't affect any non-owner user or group entries and other users would, surprisingly, still have access to the object.

  Temporarily changing a file's permission bits to no access with chmod 000 and then changing them back again was an old file locking mechanism, used before Unixes gained advisory locking mechanisms, that — as you can see — people still use even in the 21st century. (Advisory locking has been easily usable from scripts with portable well-known tools such as setlock since the late 1990s.)

• Traditional Unix scripts expect to be able to run chmod go-rwx and end up with only the object's owner able to access the object. Again, this doesn't work unless there is a mask and the old S_IRWXG permissions map to it; because otherwise that chmod command wouldn't turn off any non-owner user or group access control entries, leading to users other than the owner and non-owning groups retaining access to something that is expected to be accessible only to the owner.

  And again — as you can see — this sort of chmod command was still the received wisdom twelve years later. The rationale still holds.

• Other approaches without a mask mechanism have flaws.
• An alternative system where the permission bits were otherwise separate from and anded with the ACLs would require file permission flags to be rwxrwxrwx in most cases, which would confuse the heck out of the many Unix applications that complain when they see what they think to be world-writable stuff.
• An alternative system where the permission bits were otherwise separate from and ored with the ACLs would have the chmod(…,000) problem mentioned before.

Hence an ACL system with a mask.

Further reading

• Craig Rubin (1989-08-18). Rationale for Selecting Access Control List Features for the Unix System. NCSC-TG-020-A. DIANE Publishing. ISBN 9780788105548.
• Portable Applications Standards Committee of the IEEE Computer Society (October 1997). Draft Standard for Information Technology—Portable Operating System Interface (POSIX)—Part 1: System Application Program Interface (API)— Amendment #: Protection, Audit and Control Interfaces [C Language] IEEE 1003.1e. Draft 17.
• Winfried Trümper (1999-02-28). Summary about Posix.1e
• https://unix.stackexchange.com/a/406545/5132
• https://unix.stackexchange.com/a/235284/5132
• How can I grant owning group permissions when POSIX ACLs are applied?
• Performing atomic write operations in a file in bash