2

Im building a script to load from a file content and then use to cat and grep in another file. The problem is that is not showing any values while executing the grep. If i do the command manually it will retrieve the data.

i already use the while but the issue is the same. the output6.txt is always empty. the echo seems to work, so it seems to be the grep or cat issue.

Sample from BlockspamCLIs.txt i have the values:

4412345 
441236 
2367890

Sample All_events.csv file

1,441236,20220909120909,test
2,441237,20220909120909,test
3,441232,20220909120909,test
4,44136,20220909120909,test
5,2367890,20220909120909,test

As an output im expecting to retrieve and store and record inside of the CSV file that contains the number for example:

1,441236,20220909120909,test
5,2367890,20220909120909,test

my script:

for i in `cat BlockspamCLIs.txt`
do
   grep $i *_All_events.csv >> output6.txt

echo $i
done

thank you in advance.

Kusalananda
  • 333,661
macieira
  • 121

2 Answers2

4

First of all, please don't use a shell script for this sort of task, it is slow and error prone. In any case, grep can do what you want already:

grep -f BlockspamCLIs.txt *_All_events.csv >> output6.txt

From man grep:

       -f FILE, --file=FILE
              Obtain patterns from FILE, one per line.   If  this  option  is
              used  multiple  times  or  is  combined  with the -e (--regexp)
              option, search for all patterns given.  The empty file contains
              zero patterns, and therefore matches nothing.

Next, if you do use a shell loop to iterate over the contents of a file, never do it with for i in $(cat file). And, you must always quote your variables to avoid various problems. For example, your command won't work if any of the lines in BlockspamCLIs.txt contain whitespace.

That said, if your lines don't have spaces or shell globbing characters, the command you show would work. It would be better written like this:

while IFS= read -r i; do
    grep -e "$i" -- *_All_events.csv >> output6.txt
    printf '%s\n' "$i"
done < BlockspamCLIs.txt
Stéphane Chazelas
  • 544,893
terdon
  • 242,166
  • The issue with grep is that you can't easily restrict the match to the 2nd field only. The string 202209 would, for example, match all lines that we are given. – Kusalananda Sep 28 '22 at 08:49
  • I tried the command grep -f BlockspamCLIs.txt _All_events.csv >> output6.txt but im getting 0 values. one of the numbers there is the 145336 and manually i can find with grep 145336 _All_events.csv – macieira Sep 28 '22 at 08:57
  • 2
    @macieira are you working on Windows or can you have ever opened the BlockspamCLIs.txt in Windows? If so, that's your problem: windows line endings. Fix the file with dos2unix BlockspamCLIs.txt or sed -i 's/\r//' BlockspamCLIs.txt and try again. – terdon Sep 28 '22 at 09:02
  • this was the problem. im using Cywig to do that. i didnt realized was on windows line ending. Thank you a lot. – macieira Sep 28 '22 at 09:05
  • 1
    @macieira Good! If this solves your issue, please consider accepting the answer. Accepting an answer marks the issue as resolved. – Kusalananda Sep 28 '22 at 18:55
3

Using a tool specifically built for doing these sorts of things:

$ csvgrep -H -f BlockspamCLIs.txt -c 2 All_events.csv | tail -n +2
1,441236,20220909120909,test
5,2367890,20220909120909,test

The csvgrep utility is part of csvkit, and I use it here to extract all records from the header-less (-H) CSV file All_events.csv in which the second field (-c 2) matches any string in another file (-f BlockspamCLIs.txt).

Since the input file is a header-less CSV file, csvgrep will add a dummy header to the resulting CSV output, which we remove with tail (you could also use sed 1d or any other command that removes the first line and passes the rest on unmodified).

Since we use a CSV-aware tool, the input is not restricted to "simple" CSV data but would also work with input that contains fields with embedded commas, quotes and newlines.

This works even if the files are in DOS text format.

Kusalananda
  • 333,661