"find: paths must precede expression:" error only when run in a script

Question

If I do my find command at the command line, it works fine, but if I try it in a script, I get this error. I run the command like

FILTER=" | grep -v \"test\""
files=`find . -type f -regex \".*$ext\" $FILTER`

if I do

echo "find . -type f -regex \".*$ext\" $FILTER" 

it outputs

find . -type f -regex ".*.cpp" | grep -v "test"

which works fine at the command line. How can I get this to work in a script? I tried escaping the * too, but get the same error.

I've also noticed that

find . -type f -regex \".*$ext\"

gives no output when run in my shell script and I'm not sure why, because just like above, if I run that at command line, I get a list of .cpp files.

Answer 1

When the shell reaches the "expansions" stage, control operators (such as |) have already been identified. The result of expansions is not parsed again in search of control structures.

When the command substitution in

files=`find . -type f -regex \".*$ext\" $FILTER`

is expanded, Bash parses it as a simple command (find) followed by several arguments, two of them requiring expansion. You can turn tracing on to see the actual, expanded command:

$ set -x
$ find . -type f -regex \".*$ext\" $FILTER
+ find . -type f -regex '".*cpp"' '|' grep -v '"test"'

If you compare it with

$ set -x
$ find . -type f -regex ".*.cpp" | grep -v "test"
+ grep --color=auto -v test
+ find . -type f -regex '.*.cpp'

you can clearly see that, in the first case, the | is used as a one-character argument to find.

To dynamically build and execute a command string you need to explicitly add a new parsing stage. eval is a way to do that:

$ set -x
$ files=$(eval "find . -type f -regex \".*$ext\" $FILTER")
++ eval 'find . -type f -regex ".*cpp"  | grep -v "test"'
+++ find . -type f -regex '.*cpp'
+++ grep --color=auto -v test

But note that, when executing a variable as a script, it is really important to make sure you have control on the variable's content for obvious security reasons. Since eval tends to also make programs harder to read and to debug, it is advisable to only use it as a last resort.

In your case, a better approach could be:

filter=( -regex ".*$ext" '!' -name "*test*" )
find . -type f "${filter[@]}" -exec bash -c '
  # The part of your script that works with "files" goes here
  # "$@" holds a batch of file names
  ' mybash {} +

Which makes use of find's flexibility and also correctly handles file names that include newline characters — a corner case that makes saving the output of find into a variable unreliable, in general, unless you use something like mapfile -d '' files < <(find ... -print0) (assuming Bash (since version 4.4) and a find implementation that supports the non-standard -print0). You can read more on this in Why is looping over find's output bad practice?, also relevant in relation to piping find's output.

Again, note that the filter array's elements can cause the execution of arbitrary code (think about filter=( -exec something_evil ';' )), so you still need to make sure you have control on its content.

Answer 2

FILTER=" | grep -v \"test\""

Operators like | don't have their special meaning when they're the result of expansions. Same for quotes.

Make a function to hold the filter, the quoting is easier too:

filter() {
    grep -v "test"
}
files=$(find . -type f -regex ".*$ext" | filter)

Note that if you use find from command substitution like that, you're quashing the output to a single string, which will cause issues if you have filenames with whitespace.

---

If part of your reason to put the filter in a variable is to sometimes not have the filter, you can replace it with a null filter by redefining the function:

filter() {
    cat
}

or just making two functions, and then using a variable to decide which one to call:

filter() {
    grep -v "test"
}
filter_null() {
    cat
}
filter_used=filter
files=$(find . -type f -regex ".*$ext" | $filter_used)