Simple shell script unable to go through thousands of files; starts fine, but throws "unexpected EOF while looking for matching `"`" after some time

Question

Shell Script in Question

Let me explain what I am trying to do by e.g. so you can understand better. Let's say I have 100 .torrent files in a directory. 2 of them will download xxx.epub and yyy.epub respectively if added to a bittorrent client, but I don't know which 2 out of the 100.

So what my script does is, (1) use find to go through all .torrent files in pwd and pass each .torrent file, as it comes by, to transmission-show which will parse the .torrent file and output metadata in human readable format. We'll then use awk to get the file name the torrent file will download and run that against the list.txt which has file names we are looking for, i.e. xxx.epub and yyy.epub.

File: findtor-array.sh

#! /bin/bash
#
# Search .torrent file based on 'Name' field.
#
# USAGE:
# cd ~/myspace # location of .torrent files
# Run `findtor ~/list.txt` (if `findtor.sh` is placed in `~/bin` or `~/.local/bin`)

# Turn the list of file names from ~/list.txt (or any file passed as argument) into an array
readarray -t FILE_NAMES_TO_SEARCH < "$1"

# For each file name from the list...
for FILE_NAME in "${FILE_NAMES_TO_SEARCH[@]}"
do
    # In `pwd` and 1 directory-level under, look for .torrent files and search them for the file name
    find . -maxdepth 2 -name '*.torrent' -type f -exec bash -c "transmission-show \"\$1\" | awk '/^Name\: / || /^File\: /' | awk -F ': ' '\$2 ~ \"$FILE_NAME\" {getline; print}'" _ {} \; >> ~/torrents.txt

    # The `transmission-show` command included in `find`, on it own, for clarity:
    # transmission-show xxx.torrent | awk '/^Name: / || /^File: /' | awk -F ': ' '$2 ~ "SEARCH STRING" {getline; print}'
done

I think the process is simple and I am doing it right (except there are no checks, I know). But somehow the whole task seems too much for the script, because after running it, after sometime it starts throwing these errors continuously until I Ctrl + C it:

_: -c: line 0: unexpected EOF while looking for matching `"'
_: -c: line 1: syntax error: unexpected end of file

Are these "scaling" issues? What am I missing and what can I do to fix it?

Answer 1

FILE_NAME is being passed directly to bash -c in the -exec option of your find command. This causes problems if FILE_NAME contains quotes/shell code. In fact, arbitrary code could be executed. Example: in this particular case, the input file could contain a line '; echo "run commands";'

Instead, pass the loop var to bash -c as a positional parameter. e.g.:

find . -maxdepth 2 -name '*.torrent' -type f -exec sh -c '
transmission-show "$2" |
awk -v search="$1" '\''/^Name: / {name = substr($0,7)} /^File: / && name ~ search {print; exit}'\' \
_ "$FILE_NAME" {} \;

---

Also, it seems inefficient to loop over all search terms for each file. Consider looping over files and searching with grep -f file:

find . -maxdepth 2 -name '*.torrent' -type f -exec sh -c '
file=$1
shift
if transmission-show "$file" | head -n 1 | cut -d" " -f2- | grep -q "$@"; then
    printf "%s\n" "$file"
fi' _ {} "$@" \;

or without find:

for file in *.torrent */*.torrent; do
    if transmission-show "$file" | head -n 1 | cut -d' ' -f2- | grep -q "$@"; then
        printf '%s\n' "$file"
    fi
done

• The above simply passes all arguments to grep, so usage would be findtor -f ~/list.txt to take patterns from list, -F for fixed strings, -e expression, etc.

Answer 2

Based on suggestions from @Kusalananda, the answers (by @guest and @Jetchisel), and this detailed answer by Kevin, I came up with this:

#! /bin/bash
#
# Search for 'Name' field match in torrent metadata for all .torrent files in
# current directory and directories 1-level below.
#
# USAGE e.g.:
# cd ~/torrent-files # location of .torrent files
# Run `~/findtor.sh ~/list.txt`

# Get one file name at a time ($FILE_NAME_TO_SEARCH) to search for from list.txt
# provided as argument to this script.
while IFS= read -r FILE_NAME_TO_SEARCH; do

    # `find` .torrent files in current directory and directories 1-level under
    # it. `-print0` to print the full file name on the standard output, followed
    # by a null character (instead of the newline character that `-print` uses).
    #
    # While that's happening, we'll again use read, this time to pass one
    # .torrent file at a time (from output of `find`) to `transmission-show`
    # for the latter to output the metadata of the torrent file, followed by
    # `awk` commands to look for the file name match ($FILE_NAME_TO_SEARCH) from
    # list.txt.
    find . -maxdepth 2 -name '*.torrent' -type f -print0 |
        while IFS= read -r -d '' TORRENT_NAME; do
            transmission-show "$TORRENT_NAME" | awk '/^Name: / || /^File: /' | awk -F ': ' -v search_string="$FILE_NAME_TO_SEARCH" '$2 ~ search_string {getline; print}';
        done >> ~/torrents-found.txt

done < "$1"

I just ran this and so far it seems to be working great. So a big thank you to everyone involved!

While I did my best, any fixes and further suggestions are welcome.

Answer 3

I would write it like this.

#!/usr/bin/env bash

pattern_file="$1"

while IFS= read -r -d '' file; do
    transmission-show "$file" | awk .... "$pattern_file"   ##: Figure out how to do the awk with a file rather than looping through an array.
done < <(find . -maxdepth 2 -name '*.torrent' -type f -print0)

That should avoid the quoting hell :-)

Ok maybe the nullglob is not needed.

EDIT:

Try the find command and use it on your original script.

find . -maxdepth 2 -name '*.torrent' -type f -exec bash -c 'transmission-show "$1" | awk "/^Name\: / || /^File\: /" | awk -F ": " "\$2 ~ \"$FILE_NAME\" {getline; print}"' _ {} + >> ~/torrents.txt