Loop through all lines in a file and delete all files containing the line in their content

Question

We have a very manual and rudimentary email unsubscribe system that I am trying to automate. There is a file containing a list of the email addresses to unsubscribe, the format of this file is that there is one email address per line so cat should be able to be used I guess.

In the same folder, there are thousands of ".eml" files (raw email files) that are sent to sendmail in batch. Generating these .eml files is expensive so we keep them in a folder and send them periodically until a person unsubscribes. What I am trying to do is write a bash script looping through all the email addresses in the file, running grep on the folder for every email address, and deleting the file for which grep matches.

Since my Unix skills are very limited, I am trying to so as a reusable bash script (with loops and such) so that I get to improve my Unix skills

Answer 1

A naive approach would be (assuming GNU utilities):

grep -FZlw -f address.list -- *.eml | xargs -r0 rm -f --

Or the same but with the long options as supported by GNU utilities:

grep --fixed-strings \
     --null --files-with-matches \
     --word-regexp \
     --file address.list \
     -- *.eml |
 xargs --no-run-if-empty --null \
   rm --force --

But that would delete files when addresses are found anywhere in the file, whether it's in the From:, To:, Cc:, Reply-To headers, or in the body of the email or in attachments.

Also if the address.list contains, doe@example.com, that would also delete emails for john.doe@example.com and doe@example.com.eu.

That also assumes email addresses are formatted the same (same case, no MIME encoding) in the address.list and in the eml files.

If you know exactly how the emails are formatted, for instance if they're always going to contain one and only one occurrence of a line like:

To: address@example.com

Where address@example.com is formatted exactly like in your address.list, then you can do:

sed 's/^/To: /' address.list | grep -xZFlf - -- *.eml | xargs -r0 rm -f --

Which would be more reliable.

Instead of passing the address.list as a list of words to be found anywhere in the files, we're transforming the search list first with the stream editor command to prefix each line with "To: " so that the fixed string patterns become To: address@example.com and using -x/--line-regexp for those (instead of -w/--word-regexp) to match the full contents of lines exactly. (so To: address@example.com doesn't match on Reply-To: address@example.com.eu for instance).

Replace rm -f with grep -H '^To:' above if instead of removing the files, you want to check what the To: header is for the files that are to be removed.

Answer 2

Using the following script:

#!/bin/bash

email_dir=./emails
unsubscribe_file=./emails/unsubscribe.txt

while IFS= read -r email _; do
    files=($(grep -rni "$email" "$email_dir" | grep -v 'unsubscribe.txt'))
    if ((${#files[@]}>1)); then
        printf '%s\n' "warning: Found multiple files for: $email" "${files[@]}" >&2
    elif ((${#files[@]}==1)); then
        rm "$(echo "${files[0]}" | awk -F\: '{print $1}')"
    fi
done < "$unsubscribe_file"

---

email_dir should be set to the path to the directory containing the emails unsubscribe_file should be set to the path of the file containing the emails to be unsubscribed

The while loop will read through the unsubscribe file and for each line set the email variable to the first field (which should be the only field but _ will catch anything leftover should it exist)

We will the grep within all files of the email_dir directory for that email address (which will also return the unsubscribe file so we are using grep to remove that from our results. It would be ideal if that wasn't in the same directory. However be sure to change grep -v 'unsubscribe.txt' to reflect the actual name of your unsubscribe file)

We set these results to an array in case there is more than one result. In which case it will throw an error and remove nothing. If there is only 1 result we will extract the filename from the grep output and remove it.