problem using wildcards in filename with sed

Question

This works:

sudo sed 's/good times/bad times/' Chapter1.html > output/Chapter1.html

This does NOT work:

sudo sed 's/good times/bad times/' Chapter*.html > output/Chapter*.html

This does not work either:

sudo sed 's/good times/bad times/' *.html > output/*.html

As there are fifty chapters, can I get sed to work with wildcards?

Answer 1

As others have commented, this requires a loop, not just a wildcard.

For example, to do this with a shell for loop:

for f in ./*.html; do
  sed 's/good times/bad times/' "$f" > "output/$f"
done

This sets variable f to each .html filename in turn, executing the code inside the loop for each iteration of the loop. See below for why I used ./*.html instead of just *.html.

Notice how this has the bare-word f in the for statement itself (because that is where it is having its value set), but $f when the variable is used inside the loop (because that's where it's being expanded).

The variable expansions are also double-quoted, to ensure that they don't break the script (or worse) if they happen to contain white-space characters, or other characters with special meaning to the shell (such as ;, &, >, and many others). Failure to quote variables when they're being used is probably the number-one cause of shell scripting errors. See Why does my shell script choke on whitespace or other special characters? and $VAR vs ${VAR} and to quote or not to quote to understand why.

You can use whatever variable name you like, e.g.

for Chapter in ./*.html; do
  sed 's/good times/bad times/' "$Chapter" > "output/$Chapter"
done

---

Also worth noting: if there are no .html files in the directory, the shell will set f (or Chapter) to the literal string *.html unless you first turn on the nullglob option with shopt -s nullglob. From man bash:

nullglob

If set, bash allows patterns which match no files (see Pathname Expansion
above) to expand to a null string, rather than themselves.

---

BTW, I used ./*.html with the for loop instead of just *.html in order to protect against filenames that sed might otherwise interpret as one of its command-line options.

As @StéphaneChazelas mentioned in a comment, if a filename starting with -e and ending with #.html were in the directory, sed would interpret that as a sed script to be executed. This is unlikely (but excrement occurs, as does malice) but it's good to program defensively as much as possible.

By using ./*.html, instead of sed seeing an argument of, e.g., -e1,$d due to a file named -e1,$d#.html (which is a completely valid filename), it sees an argument of./-e1,$d which is not going to be interpreted as one of sed's command line options...sed's options don't start with ./.

Also: because $f starts with ./, the output for a filename like foo.html will be redirected to output/./foo.html. This is perfectly fine, having extra ./ elements in a path still resolves to the same destination. Even something absurd like output/./././[a million more ./s]/foo.html is still just output/foo.html

If you are using GNU sed (which is the standard sed on linux) or (almost?) any modern version of sed, you can use -- to indicate the end of option arguments instead:

for f in *.html; do
  sed 's/good times/bad times/' -- "$f" > "output/$f"
done

or do both:

for f in ./*.html; do
  sed 's/good times/bad times/' -- "$f" > "output/$f"
done

Answer 2

The shell expands wildcards on the command line, and the command being run neither sees them nor knows what to do with them.

So if you had the files a.html b.html and output/b.html output/c.html and you ran

sed ... *.html > output/*.html

the command that would actually run would be

sed ... a.html b.html > output/b.html output/c.html

which is a syntax error (can't redirect to two files) and nothing like what you probably intended.

The solution here is to use a for loop and replace the * with the loop's index variable. Some quoting will be necessary if there are spaces in any of the filenames. There are plenty of examples of how to do this right in the duplicates for this question.

Answer 3

Assuming you are in the working directory that contains the files, you could use find

$ find . -name 'Chapter*' -exec sed 's/good times/bad times/woutput/{}' {} 2> /dev/null \;