linux + delete words from file that appear in another file

Question

I have two files: in linux redhat version 6

list1.txt - this files contain 34732 words

list2.txt - this files contain 272 words

I want to remove all words from list1.txt that spears from list2.txt

what the best approach to do that ( both files include only one field )

more list1.txt

dweferf
fr
grgr
gefyh
fergtrg
ggtgg
fergth
gtg
.
.
.

more list2.txt

dweferf
fr
frgrgggb
rggtgtrgrt
fergtrg
rfergrtg
fwed4

.
.
.

I mean that all words/string in list1.txt that match the list in list2.txt should be deleted from list1.txt , so list1.txt will not include the words that in list2.txt — yael, Jan 24 '17 at 11:10
egrep -v -f file2 file1 Where file2 are the things you want removed from file1 — Essex Boy, Jan 24 '17 at 11:23
grep -Fxvf list2.txt list1.txt would be better option.. will ensure that for ex: fr in list2.txt won't match free in list1.txt... -F since we don't need regex here, -x to match only whole line — Sundeep, Jan 24 '17 at 11:33
if order is not important, comm -23 <(sort list1.txt) <(sort list2.txt) — Sundeep, Jan 24 '17 at 11:34
possible duplicate http://unix.stackexchange.com/q/28158/200853 — Wissam Roujoulah, Jan 24 '17 at 11:51

score -1 · Answer 1 · answered Jan 24 '17 at 11:39

-1

You can do something like this too,

#!/bin/bash
FILE1=$1 # path to your list2.txt
FILE2=$2 # path to your list1.txt
touch ./tmpfile
FILE3=./tmpfile
while read p; do
  if [ `grep "\<$p\>" $FILE2 | wc -l` -ne 0 ] 
  then
    echo "$p word found discarding"
  else
    echo -e "$p" >> $FILE3
  fi
done <$FILE1
mv $FILE3 $FILE2

answered Jan 24 '17 at 11:39

Fahad Siddiqui

101

Probably the slowest approach conceivable, or pretty close to it. – Satō Katsura Jan 24 '17 at 11:42
yeah you are right the computational complexity goes to O(N^2). but this works as well. – Fahad Siddiqui Jan 24 '17 at 11:43
1

@Fahad Too much unnecessary code. Can be done with a single line - one single grep as advised in OP comments. – George Vasiliou Jan 24 '17 at 12:29
happy learning! – Fahad Siddiqui Jan 24 '17 at 13:11

linux + delete words from file that appear in another file

1 Answers1