How to loop over command result while still running

Question

I have a script that uses find to search for files and loops over the result for further processing.

for file in $(find ~ -type f -size +1G); do
     filename=$(basename $file)
     echo $file
done

When I run the find command directly, it outputs matches directly. But its whin this script, the loop first waits for the find command to completely finish and afterwards loops over the result.

Is it possible to have the loop run over a match as soon as the find command finds any match?

It would be possible with using pipe the output, but I want to do more complex stuff in the loop afterwards than simple one line stuff.

The for loop will not start until its arg list is complete. You could change the construct to find ... | while IFS='' read -r FN; do but this can bring its own issues. If (as suggested) you can integrate the processing into -exec or | xargs, it would be more stable. — Paul_Pedant, Dec 05 '20 at 20:10

αғsнιη · Answer 1 · 2020-12-05T20:06:55.950

Learn to use in this way instead:

find ~ -type f -size +1G -exec bash -c 'do-stuff-with "$1"' bash_find {} \;

Within find's -exec ... {} \; construct we open a inline script with bash -c '...' that find command will run this for every file it finds.

bash_find is the name of our inline script which represent in $0 for our script and in case of any error stopped the script it will be used to give us a visual output that where the failure come from with printing bash_find followed by short description of the error.

thanasisp · Answer 2 · 2020-12-05T20:40:46.023

See BashFAQ/024 which is relative to the scope of variables and other stuff, depending on what you would use. Also expands on portability across shells. Below examples are for bash and GNU null separation.

With process substitution and redirecting to while loop and null separation for the arguments:

count=0
while IFS= read -r -d $'\0' f; do
    printf "%s Processing file: %s\n" "$((++count))" "$f" 
done < <(find -type f -print0)

With command grouping:

count=0
find -type f -print0 |
    {
        while IFS= read -r -d $'\0' f; do
            printf "%s Processing file: %s\n" "$((++count))" "$f" 
        done
    }

I have added a variable count to demonstrate that for both cases the scope is convenient, you can use into the while loop whatever defined earlier. For the second case, anything introduced into the grouping, will not be accesible afterwards for your script. You can test it by defining a new variable into the grouping and trying to access it later, it will be empty. This is probably a reason to prefer the first way for some cases.

How to loop over command result while still running

2 Answers2