Is it even possible to correctly handle all possible file names?

Question

In linux only two characters are forbidden in filenames "slash" and "null character". So every character with special meaning in every script language should be escaped, BUT every escape sequence is allowed too in file names! Even worse that i.e. bash some escaping methods escapes only some characters, so to escape large amount of different characters you should use a couple of different escaping methods together, BUT they interfere with each other! Even worse that some commands use some characters for their purposes, and other commands use others, so for every single simple operation on files you should escape file name differently! Even worse that only null character could be used to separate filenames safely, but most commands cannot work with that. Even worse, that in linux basically everything is file... So this seems not only nuisance, but matter of security and stability because large portion of linux is script-based so very flawed!

So show me where i'm wrong... Is it even possible to correctly handle all possible file names?

Clarification. Originally i wanted to:

1. list files and folders under given path

2. search list to find ones match to given criteria (age or file pattern or size)

3. move matched files and folders to categories i.e. movies Because of complexity of tests it was not possible (or practical) to do it in one command, so i had to pass file name between different commands. Bash globbing was first thing to throw off because of spaces in filenames. Globbing always split filename with spaces to two elements of list. Then i tried use "find". This was better, but much slower, and difficult to use.

I cannot use any special character to escape file name, because i don't know what character might be in file name. After some testing i discovered that is matter of time before any character will occur.

I've tried defined filter like: audio_ext=(*.mp3 *.wav *.ogg *.mid *.mod *.stm *.s3m *.it *.wma *.669 *.ac3) Soon i've realized that this way i cannot define filters for multiple uses, because globbing kicks rigths away. So i've disabled globbing and history by set -fH. Without globbing i had to do expansion by hand

while IFS= read -r -d $'\0'; do list+=("$REPLY") done < <( find . -maxdepth 1 -mindepth 1 ${params[@]} -print0 2>/dev/null )

Where params is array like "-iname" "*.mp3" "-o" "-iname" "*.wav" etc. This worked until file had "(" in name. Find returned error about wrong usage.

To tell the truth... I've used batch script for this task until recently for 15 years. Time spend on writing was around one or two afternoons. It had drawbacks and issue with ! in filenames, but generally it worked. Now i have trying almost two months to write it in bash. It's ugly, complicated, very buggy, and it seems it will never work good.

Answer 1

Simple. Use globbing to select the files you want, and quote the variable that holds the filename:

shopt -s nullglob
for file in ./*.txt; do
    do_something_with "$file"
done

That's really all there is to it.

More details:

• http://mywiki.wooledge.org/ParsingLs
• https://www.dwheeler.com/essays/filenames-in-shell.html

---

Update: globbing is not responsible for the word splitting effect you're seeing. Failing to quote the variable is.

You can get file info for your conditions with stat

read size mtime < <(stat -c "%s %Y" "$file")
[[ $size -gt 1000 ]] && echo "too big"
[[ $mtime -lt $(date -d yesterday +%s) ]] && echo "too old"

---

Update 2: creating a filename with many special characters in it requires mixing various quoting mechanisms, but it's still possible to do anything with that file.

$ filename='~ASDFzxcv!@#$%^&*()_+[]\{}|;:",.<>?`'"'"$' \a\t\n\r\f'".txt"
#          ^^ single quoted part ^^^^^^^^^^^^^^^^   
#                             double quoted part ^^^
#                                ANSI-C quoted part ^^^^^^^^^^^^^^

$ echo "$filename"
~ASDFzxcv!@#$%^&*()_+[]\{}|;:",.<>?`'   

.txt

$ printf "%q\n" "$filename"
$'~ASDFzxcv!@#$%^&*()_+[]\\{}|;:",.<>?`\' \a\t\n\r\f.txt'

$ date > "$filename"

$ cat "$filename"
Thu Apr 12 15:14:29 EDT 2018

$ ls -lt
total 3836
-rw-rw-r-- 1 jackman jackman      29 Apr 12 15:14 ~ASDFzxcv!@#$%^&*()_+[]\{}|;:",.<>?`' ?????.txt
                ︙

$ ls -lt --show-control-chars
total 3836
-rw-rw-r-- 1 jackman jackman      29 Apr 12 15:14 ~ASDFzxcv!@#$%^&*()_+[]\{}|;:",.<>?`'     

.txt
                ︙

If the output of ls is redirected to anything other than a terminal (e.g., a file or a pipe), it will use the --show-control-chars style by default.  You can see this by running ls -lt | cat.  ls has other display options; e.g., --quoting-style=WORD.

Answer 2

Filenames can use any character except the nul character (\0) and slash, which is a path separator. Variables may hold any data (except nul characters in most shells). If properly quoted, filenames may be safely stored in variables and used with utilities.

Regarding your points:

To iterate over a set of files (regular files or directories), you may use a simple shell loop like

for name in ./*; do
    # some code that uses "$name"
done

To iterate over files while selecting particular files using specific criteria, find is a better choice. for example, to select all regular files in the current directory (or below) that are older than N days (have modification date at least N days in the past):

find . -type f -mtime +N

Similarly, -size is used to select files based on size, and -name to match the filename against a globbing pattern.

For example, to select the regular files that have filenames that match *.mov and that have been modified in the last week:

find . -type f -name '*.mov' -mtime -7

Then, to actually do something with these files, like moving them to the $HOME/Movies directory:

find . -type f -name '*.mov' -mtime -7 -exec mv {} "$HOME/Movies" ';'

The {} will be replaced by the pathname of the file in the invocation of mv. You do not need to quote the {} (it won't change anything if you do), as find will not invoke the shell's word splitting or filename expansion on the pathname.

A further improvement to this would be to detect file name collisions in the destination directory. For this we use a short helper script that will take a number of filenames on its command line:

destdir="$HOME/Movies"
for name do
    if [ -f "$destdir/${name##*/}" ]; then
        printf "%s already exists in %s, not overwriting it!\n" "${name##*/}" "$destdir" >&2
    else
        mv "$name" "$destdir"
    fi
done

or, in shortcut form:

destdir="$HOME/Movies"
for name do
    [ -f "$destdir/${name##*/}" ] && printf "skipping %s\n" "$name" >&2 && continue
    mv "$name" "$destdir"
done

Plugging that into our find command:

find . -type f -name '*.mov' -mtime -7 -exec sh -c '
    destdir="$HOME/Movies"
    for name do
        [ -f "$destdir/${name##*/}" ] && printf "skipping %s\n" "$name" >&2 && continue
        mv "$name" "$destdir"
    done' sh {} +

Nowhere along the way do we allow the shell to do word splitting or filename globbing on the pathname or filename that we are currently processing.

For further information:

• Why is looping over find's output bad practice?
• Security implications of forgetting to quote a variable in bash/POSIX shells
• Why does my shell script choke on whitespace or other special characters?
• Why *not* parse `ls`?
• Is it possible to use `find -exec sh -c` safely?