2

I am trying to copy lines from source.txt to target.txt. I would like this bash script to check each line on target.txt if there is duplicate entry before copy.

source.txt contains:

a$$a$$a
b**b**
c%%cc%%
d##d##d##
e^^e^^e^^

target.txt contains:

a$$a$$a
ee$$ee$$
ff__ff__
gg@@gg@@
zzxxzzxx
bb..bb..bb
e^^e^^e^^
hh;;hh;;hh

on this scenario, I assuming that there only 3 entries will be copied to target.txt which are:

b**b**
c%%cc%%
d##d##d##

My test code is:

#!/bin/bash
echo "started"
programpath=/home/mysite/www/copyfiles

var str input ; cat "$programpath/source.txt" > $input 
var str target ; cat "$programpath/target.txt" > $target 

cat $input >> $target

uniq -u "$target"

echo "finished"
    exit 1
fi
αғsнιη
  • 41,407
danone
  • 304

1 Answers1

1

Why using bash? The grep command can do the job clean.

grep -Fxvf target.txt source.txt #>> target.txt

This will return those lines which are exist only in source.txt, then you can append these lines to your target.txt with just uncomment #>> target.txt.

You may also needs to unique the source.txt before, to prevent appending duplicated entries if in source.txt file which awk also does the same at next.

grep -Fxvf target.txt <(sort -u source.txt) #>> target.txt
  • The -F option is telling grep that match pattern as a string instead of regex.
  • With -x option we are telling whole line is my pattern.
  • The -v is the reverse match that if you miss it, that will output the lines which are exist in the both files.
  • And -f is telling grep read my patterns from a file which is target.txt here.

Or you could use awk instead.

awk 'NR==FNR{seen[$0]=1;next} !seen[$0]++' target.txt source.txt #>> target.txt

  • Add whole target.txt file into the array called seen with the key of whole line seen[$0], and do next to read next line.

  • With !seen[$0]++ we are looking a line from source.txt which is not exist in array, then print it. Also add source.txt file lines into the array to prevent printing duplicated lines if exist in source.txt_.

αғsнιη
  • 41,407