Why does my find (+ sed) command work in the terminal but not in a makefile?

Question

I have the following command:

find stdlib/main -type f -exec sh -c "echo {} | sed -e 's/stdlib\/main\///g' -e 's/\.q//g' -e 's/\//\./g' -e 's~^~/resource:\"{},~g' -e 's/$/\"/g'" \;

The goal is to find all files in stdlib/main (and subdirectories) and format them as such: {filename},{filename-with-stdlibmain-removed-and-extension-removed-and-slashes-changed-to-dots}

The command works perfectly when I run it myself. But I am trying to use it in a makefile as such:

STDLIB_RESOURCES=$(shell find stdlib/main -type f -exec sh -c "echo {} | sed -e 's/stdlib\/main\///g' -e 's/\.neo//g' -e 's/\//\./g' -e 's~^~/resource:\"{},~g' -e 's/$/\"/g'" \;)

When I run the makefile, I get one of these errors per file found:

sed: -e expression #5, char 5: unterminated `s' command

What am I missing here?

What are you missing? Probably the fact that you have quotes within quotes within quotes, and that $(…) is adding one more level of containment. A workaround would be to put the sed commands into a file and invoke it with sed -f. I don’t immediately know how to do it in a self-contained way. — G-Man Says 'Reinstate Monica', Apr 12 '19 at 00:15
There's gotta be some way to do it in a self-contained way... — Matthew Peterson, Apr 12 '19 at 00:16
Try removing just the last sed expression (-e 's/$/\"/g'). This is the most likely to be tripping you up as it contains a " character. If that works, then you probably need to escape it better. — Crypteya, Apr 12 '19 at 00:58
I get this when I do that: /bin/sh: -c: line 0: unexpected EOF while looking for matching /bin/sh: -c: line 0: unexpected EOF while looking for matching `"'' /bin/sh: -c: line 1: syntax error: unexpected end of file — Matthew Peterson, Apr 12 '19 at 02:31
Checked again and both of the last two sed expressions have " characters. Remove both of them. If that returns successfully, look at how to escape better. Your new error message definitely makes it look like the " that you removed was being interpreted before it could count in the sed command — Crypteya, Apr 12 '19 at 05:06

score 1 · Accepted Answer · answered Apr 12 '19 at 05:56

The main thing you are missing is that $ is a special character to make, and quoting is different in Make and the shell.

So for example

's/$/\"/g'

the `` pair protect everything inside them to the shell (and incidentally makes the \ unnecessary), but doesn't to make, so to make it looks like

 's/\"/g'

assuming you don't have a variable called / (which you could in make, but typically not in the shell).

The first thing to do is to replace $ with $$.

Kusalananda · Answer 2 · 2019-04-12T07:11:10.440

Using {} in an in-line script that you execute through find is a code injection vulnerability. Don't do that. The first argument to sh -c (the script) should be single quoted and the arguments to that script should be passed on its command line.

Instead, write the find command as something like the following (uses bash instead of sh to be able to use ${parameter//pattern/word} in one place):

find stdlib/main -type f -exec bash -c '
    for pathname do
        string=${pathname#stdlib/main/} # delete initial path
        string=${string%.*}             # delete suffix after last dot
        string=${string////.}           # change slashes to dots

        # output:
        printf "resource:\"%s,%s\"\n" "$pathname" "$string"
    done' bash {} +

Instead of using sed, this uses parameter substitutions to modify the pathnames found by find. The in-line bash script will be executed for batches of found files and will iterate over the pathnames in each batch. The printf would output the data transformed in the same way as what your sed command is doing (if I managed to decipher it correctly, it wasn't just what you described).

How you later cope with filenames containing double quotes and commas is another issue (the output string after resource: would potentially be difficult to parse).

It would be easiest to put the find command into a separate script and call that from GNU make in $(shell ...), or you'll end up with something like

STDLIB_RESOURCES := $(shell     \
find stdlib/main -type f -exec bash -c '    \
    for p do                                \
        s=$${p\#stdlib/main/};              \
        s=$${s%.*};                         \
        s=$${s////.};                       \
        printf "resource:\"%s,%s\"\n" "$$p" "$$s"; \
    done' bash {} + )

in your Makefile (due to the way GNU make handles variables etc.) Also note the :=. You need this for the command to be executed immediately upon assigning to the variable, not every time the variable is accessed.

Congrats on becoming a moderator :) I put you as #1 when I voted. — Crypteya, Apr 12 '19 at 05:59

Why does my find (+ sed) command work in the terminal but not in a makefile?

2 Answers2