How to find all files containing various strings from a long list of string combinations?

Question

I am still very new to command line tools (using my Mac OSX terminal) and hope I haven't missed the answer somewhere else, but I have searched for hours.

I have a text file (let's call it strings.txt) containing 200 combinations of 3 strings. [Edit 2017/01/30] The first five rows look like this:

"surveillance data" "surveillance technology" "cctv camera"
"social media" "surveillance techniques" "enforcement agencies"
"social control" "surveillance camera" "social security"
"surveillance data" "security guards" "social networking"
"surveillance mechanisms" "cctv surveillance" "contemporary surveillance"

Note that I can change strings.txt to any other format, as long as the bigrams/ 2-word phrases like surveillance data in line 1 stay together. (That means I can delete the quotes if necessary, as for the answer by @MichaelVehrs below).

Now I want to search a directory of more than 800 files for those files that contain at least one of the string combinations (anywhere in the file). My original idea was to use egrep with a pattern file like this:

egrep -i -l -r -f strings.txt file_directory

However, I can only get this to work if there is one string per line. This is not desirable, because I need the identified files to contain all three strings of a given pattern. Is there a way to add some kind of AND operator to the grep pattern file? Or is there another way to achieve what I want using another function/tool? Many thanks!

Edit 2017/01/30

The answer by @MichaelVehrs below was very helpful; I edited it to the following:

while read one two three four five six
do grep -ilFr "$one $two" *files* | xargs grep -ilFr "$three $four" |  xargs grep -ilFr "$five $six"
done < *patternfile* | sort -u

This answer works when the pattern file contains the strings without quotes. Sadly, it only seems to match the pattern on the first line of the pattern file. Does anyone know why?

Edit 2017/01/29

A similar question about grepping multiple values has been asked before, but I need the AND logic in order to match one of the three-string-combinations from the pattern file strings.txt in the other files. I realise that the format of strings.txt might have to be changed for the matching to work and would appreciate suggestions.

Answer 1

I'd use perl, something like:

perl -MFile::Find -MClone=clone -lne '
  # parse the strings.txt input, here looking for the sequences of
  # 0 or more characters (.*?) in between two " characters
  for (/"(.*?)"/g) {
    # @needle is an array of associative arrays whose keys
    # are the "strings" for each line.
    $needle[$n]{$_} = undef;
  }
  $n++;

  END{
    sub wanted {
      return unless -f; # only regular files
      my $needle_clone = clone(\@needle);
      if (open FILE, "<", $_) {
        LINE: while (<FILE>) {
          # read the file line by line
          for (my $i = 0; $i < $n; $i++) {
            for my $s (keys %{$needle_clone->[$i]}) {
              if (index($_, $s)>=0) {
                # if the string is found, we delete it from the associative
                # array.
                delete $needle_clone->[$i]{$s};
                unless (%{$needle_clone->[$i]}) {
                  # if the associative array is empty, that means we have
                  # found all the strings for that $i, that means we can
                  # stop processing, and the file matches
                  print $File::Find::name;
                  last LINE;
                }
              }
            }
          }
        }
        close FILE;
      }
    }
    find(\&wanted, ".")
  }' /path/to/strings.txt

That means we minimize the number of string searches.

Here, we're processing the files line by line. If the files are reasonably small, you could process them as a whole which would simplify it a bit and might improve performance.

Note that it does expect the list file to be in the:

 "surveillance data" "surveillance technology" "cctv camera"
 "social media" "surveillance techniques" "enforcement agencies"
 "social control" "surveillance camera" "social security"
 "surveillance data" "security guards" "social networking"
 "surveillance mechanisms" "cctv surveillance" "contemporary surveillance"

format, with a number (doesn't have to be 3) of quoted (with double quote) strings on each line. The quoted strings cannot contain double quote characters themselves. The double quote character is not part of the text being searched. That is if the list file contained:

"A" "B"
"1" "2" "3"

that would report the path of all the regular files in the current directory and below that contain either

• both A and B
• or (being not an exclusive or) all of 1, 2 and 3

anywhere in them.

Answer 2

The problem is a bit awkward, but you could approach it like this:

while read one two three four five six
  do grep -lF "$one $two" *files* | xargs grep -lF "$three $four" | xargs grep -lF "$five $six"
done < patterns | sort -u

This assumes that your pattern file contains exactly six words per line (three patterns of two words each). The logical and is realized by chaining three consecutive filters (grep). Note that this is not particularly efficient. An awk solution would probably be faster.

Answer 3

This is another approach that seems to work in my tests.

I copied your strings file data to a file named d1.txt and move it to a separate directory (i.e tmp) to avoid later grep to match strings file in the same file (d1.txt).

Then insert in this strings file (d1.txt in my case) a semi colon between each search term with the following command: sed -i 's/" "/";"/g' ./tmp/d1.txt

$ cat ./tmp/d1.txt
"surveillance data" "surveillance technology" "cctv camera"
"social media" "surveillance techniques" "enforcement agencies"
"social control" "surveillance camera" "social security"
"surveillance data" "security guards" "social networking"
"surveillance mechanisms" "cctv surveillance" "contemporary surveillance"
$ sed -i 's/" "/";"/g' ./tmp/d1.txt
$ cat ./tmp/d1.txt
"surveillance data";"surveillance technology";"cctv camera"
"social media";"surveillance techniques";"enforcement agencies"
"social control";"surveillance camera";"social security"
"surveillance data";"security guards";"social networking"
"surveillance mechanisms";"cctv surveillance";"contemporary surveillance"

Then remove the double quotes using command sed 's/"//g' ./tmp/d1.txt PS: This may no be really necessary, but i removed double quotes for testing.

$ sed -i 's/"//g' ./tmp/d1.txt && cat ./tmp/d1.txt
surveillance data;surveillance technology;cctv camera
social media;surveillance techniques;enforcement agencies
social control;surveillance camera;social security
surveillance data;security guards;social networking
surveillance mechanisms;cctv surveillance;contemporary surveillance

No you can grep all files in current directory with the program agrep which is designed exactly to provide multi pattern grep with AND operation.

agrep requires multiple patterns to be separated by semi colon ; in order to be evaluated as AND.

In my tests, i created two sample files with contents:

$ cat d2.txt
This guys over there have the required surveillance technology to do the job.

The other guys not only have efficient surveillance technology, but they also gather surveillance data by one cctv camera.

$ cat d3.txt
All surveillance data are locked.
All surveillance data are locked and guarded by security guards.
There are several surveillance mechanisms (i.e cctv surveillance, contemporary surveillance, etv)

Running agrep on the current directory returns the correct lines (with AND) and filenames:

$ while IFS= read -r line;do agrep "$line" *;done<./tmp/d1.txt
d2.txt: The other guys not only have efficient surveillance technology, but they also gather surveillance data by one cctv camera.
d3.txt: There are several surveillance mechanisms (i.e cctv surveillance, contemporary surveillance, etv)

Answer 4

Since agrep seems not to be present in your system, have a look in this alternative based on sed and awk to apply grep with and operation from patterns read by a local file.

PS: Since you use osx i'm not sure if the awk version you have will support bellow usage.

awk can simulate grep with AND operation of multiple patterns in this usage:
awk '/pattern1/ && /pattern2/ && /pattern3/'

So you could transform your pattern file from this:

$ cat ./tmp/d1.txt
"surveillance data" "surveillance technology" "cctv camera"
"social media" "surveillance techniques" "enforcement agencies"
"social control" "surveillance camera" "social security"
"surveillance data" "security guards" "social networking"
"surveillance mechanisms" "cctv surveillance" "contemporary surveillance"

To this:

$ sed 's/" "/\/ \&\& \//g; s/^"/\//g; s/"$/\//g' ./tmp/d1.txt
/surveillance data/ && /surveillance technology/ && /cctv camera/
/social media/ && /surveillance techniques/ && /enforcement agencies/
/social control/ && /surveillance camera/ && /social security/
/surveillance data/ && /security guards/ && /social networking/
/surveillance mechanisms/ && /cctv surveillance/ && /contemporary surveillance/

PS: You can redirect the output to another file by using >anotherfile in the end , or you can use the sed -i option to make in-place changes in the same search terms pattern file.

Then you just need to feed awk with awk-formatted patterns from this pattern file :

$ while IFS= read -r line;do awk "$line" *.txt;done<./tmp/d1.txt #d1.txt = my test pattern file

You could also not transform patterns in your original pattern file by applying sed in each line of this original pattern file like this:

while IFS= read -r line;do 
  line=$(sed 's/" "/\/ \&\& \//g; s/^"/\//g; s/"$/\//g' <<<"$line")
  awk "$line" *.txt
done <./tmp/d1.txt

Or as one-liner:

$ while IFS= read -r line;do line=$(sed 's/" "/\/ \&\& \//g; s/^"/\//g; s/"$/\//g' <<<"$line"); awk "$line" *.txt;done <./tmp/d1.txt

Above commands return the correct AND results in my test files that look like this:

$ cat d2.txt
This guys over there have the required surveillance technology to do the job.
The other guys not only have efficient surveillance technology, but they also gather surveillance data by one cctv camera.

$ cat d3.txt
All surveillance data are locked.
All surveillance data are locked and guarded by security guards.
There are several surveillance mechanisms (i.e cctv surveillance, contemporary surveillance, etv)

Results:

$ while IFS= read -r line;do awk "$line" *.txt;done<./tmp/d1.txt
#or while IFS= read -r line;do line=$(sed 's/" "/\/ \&\& \//g; s/^"/\//g; s/"$/\//g' <<<"$line"); awk "$line" *.txt;done <./tmp/d1.txt
The other guys not only have efficient surveillance technology, but they also gather surveillance data by one cctv camera.
There are several surveillance mechanisms (i.e cctv surveillance, contemporary surveillance, etv)

Update:
Above awk solution prints the contents of matching txt files.
If you want to display the filenames instead of the contents, then use the following awk where necessary:

awk "$line""{print FILENAME}" *.txt