Shell for loop through file and for each line use sed to delete lines from other file

Question

I'm trying to clean up long-cluttered-filename.txt by deleting the lines that match the strings in my delete-these-lines.txt...

$ cat delete-these-lines.txt
context
platform
server
civicrm
site_enabled
client
redirection
cron_key
^\s*[0-9]*\s
alias
profile
install
ssl_
address

...and with a simple for loop I can iterate through the lines of that file like so:

$ for l in delete-these-lines.txt; do cat $l; done
context
platform
server
civicrm
site_enabled
client
redirection
cron_key
^\s*[0-9]*\s
alias
profile
install
ssl_
address

but when I try to substitute cat for sed it doesn't work. Neither of these work:

for l in delete-these-lines.txt; do sed -i "/$l/d" long-cluttered-filename.txt; done
for l in delete-these-lines.txt; do sed -i '/"$l"/d' long-cluttered-filename.txt; done

Is this a problem with the $l variable being interpolated? Is this a limitation of for loops?

I see answers like loop sed to delete line with strings provided in a list of strings from a text file and Read line from file then delete that use while loops with read but I'd prefer to use a for loop if it's possible, and if it's not possible I'd like to know why.

for l in delete-these-lines.txt ... loops over the filename, once. You presumably want for l in $(<delete-these-lines.txt) ... or for l in $(cat delete-these-lines.txt) ... to loop over the file's contents. But it's a bad idea. — steeldriver, Feb 13 '20 at 18:10
OK perhaps I shouldn't have said "bad idea" - it's just hard to see why you are so determined to use a for loop in this context. — steeldriver, Feb 13 '20 at 18:44
Does delete-these-lines.txt contain patterns or strings? That is, should ^\s*[0-9]*\s match that string literally, or is that a PCRE regular expression? — Kusalananda, Feb 13 '20 at 23:00
@Kusalananda ^\s*[0-9]*\s is a regex pattern (I'm not sure about PCRE though), not a literal string. — alec, Feb 14 '20 at 00:26
@steeldriver when you say "bad idea" are you thinking something along the lines of using a shell loop to process text considered bad practice? I'm not as determined to "use" a for loop as I am to figure out if and how it's possible; the grep -v -f.... option looks great in how it functions, but I'm trying learn about how these loops work... and your comment on the distinction between looping over the filename vs the file contents was helpful :) — alec, Feb 14 '20 at 00:41
I was thinking more specifically about the issue of word splitting, as described here How to loop over the lines of a file? and explored more generically in Why does my shell script choke on whitespace or other special characters?. If you can guarantee that your lines contain no whitespace, you may not care. — steeldriver, Feb 14 '20 at 00:47

score 1 · Answer 1 · answered Feb 13 '20 at 19:14

If you have two files where one contains list of strings to delete from another one, you can do the entire thing with one simple grep:

grep -v -f delete-these-lines.txt long-cluttered-filename.txt

This will output all the strings from long-cluttered-filename.txt which do not match to patterns from delete-these-lines.txt.

Note, that in case some string in long-cluttered-filename.txt will contain client (e.g. 123superclient456) it still matches to client and will be excluded from the output.

score 1 · Answer 2 · answered Feb 15 '20 at 00:04

Never use the letter l as a variable name as it looks far too much like the number 1 and so obfuscates your code. What you want to do is a bad idea as you already know, but see https://mywiki.wooledge.org/DontReadLinesWithFor for how to read lines from a file in shell and then you'd write it as:

while IFS= read -r regexp; do
    sed -i "/$regexp/d" long-cluttered-filename.txt
done < delete-these-lines.txt

but again - don't do it for all the reasons you've already been warned of (plus you're just re-implementing grep -v -f delete-these-lines.txt long-cluttered-filename.txt > tmp && mv tmp long-cluttered-filename.txt less efficiently, less robustly, and less portably).

Shell *for* loop through file and for each line use sed to delete lines from other file

2 Answers2

Shell for loop through file and for each line use sed to delete lines from other file