grep multi text file input and create diferent output each file

Question

i have script that filter several text file using grep and awk in loop, my issue is when create output to each file after filtered,

this my script:

grep_multi.sh

path=$(find /home/folder/file/source -iname "Tracert*" )

for i in "$path"
do
        grep -E '^%%.*.%%$'\|'IPv4 type' $i | awk '/%%TRACERT:/ {sfx = $0; next} {print $1","$2","$3","$4","$5","$6","$7" "$8","sfx}' > filter.result.$i 

done

when running the script i got error like this:

./grep_multi.sh: line 5: filter.result.$i: ambiguous redirect

this for variabel $path

$ find /home/folder/file/source -iname "Tracert*"
/home/folder/file/source/Tracert_1.txt
/home/folder/file/source/Tracert_2.txt
/home/folder/file/source/Tracert_3.txt
/home/folder/file/source/Tracert_4.txt
/home/folder/file/source/Tracert_5.txt
/home/folder/file/source/Tracert_6.txt
/home/folder/file/source/Tracert_7.txt
/home/folder/file/source/Tracert_8.txt

tracert_1.txt

O&M    #108
%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%
RETCODE = 0  Operation succeeded

The result is as follows
------------------------
 Record index  Response number  First response time(ms)  Second response time(ms)  Third response time(ms)  IP type    Peer IP address

 1             3                1                        1                         1                        IPv4 type  10.10.10.10    
 2             3                1                        1                         1                        IPv4 type  10.10.10.10  
 3             0                NULL                     NULL                      NULL                     IPv4 type  Timeout        
 4             0                NULL                     NULL                      NULL                     IPv4 type  Timeout        
 5             3                1                        1                         1                        IPv4 type  10.10.10.10   
 6             3                1                        1                         1                        IPv4 type  10.10.10.10   
 7             3                1                        1                         1                        IPv4 type  10.10.10.10

By quoting the variable "$path" you are causing the loop to run once, with $i expanding to the whole list of paths. See this somewhat related question Why is looping over find's output bad practice? — steeldriver, Oct 16 '19 at 14:14
^ what he says. also: that grep expression looks "wrong" (%%.*.%% => %%.*%%, unsure why we close a quoted block, escape a pipe then open a new quote). Could you give us a sample of what you're trying to extract? — SYN, Oct 16 '19 at 14:30
@SYN i already update my post tracert_1.txt, i want extract every %%TRACERT line fill every "hop", so i can make csv file from that — ddrackochan, Oct 16 '19 at 14:56
allright, makes much more sense now. Should be able to simplify your grep, with: grep -E '%%.*%%$|IPv4 type' $i ... — SYN, Oct 16 '19 at 15:07

steeldriver · Accepted Answer · 2019-10-16T16:24:21.577

By quoting the variable "$path" you are causing the loop to run once, with $i expanding to the whole list of paths. So your redirection ends up something like

> filter.result./home/folder/file/source/Tracert_1.txt /home/folder/file/source/Tracert_2.txt ...

which is "ambiguous". See this somewhat related question Why is looping over find's output bad practice?

You don't really need a shell loop - and you don't need grep either. You can select the IPv4 lines and redirect to a file whose name is derived from the current FILENAME all using awk:

awk '
  /%%TRACERT:/ {sfx = $0; next} 
  /IPv4 type/ {print $1","$2","$3","$4","$5","$6","$7" "$8","sfx > "filter.result." FILENAME}
' tracert_*.txt

For local files, this will produce outputs like:

$ head filter.result*
==> filter.result.tracert_1.txt <==
1,3,1,1,1,IPv4,type 10.10.10.10,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%
2,3,1,1,1,IPv4,type 10.10.10.10,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%
3,0,NULL,NULL,NULL,IPv4,type Timeout,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%
4,0,NULL,NULL,NULL,IPv4,type Timeout,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%
5,3,1,1,1,IPv4,type 10.10.10.10,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%
6,3,1,1,1,IPv4,type 10.10.10.10,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%
7,3,1,1,1,IPv4,type 10.10.10.10,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%

==> filter.result.tracert_2.txt <==
1,3,1,1,1,IPv4,type 10.10.10.10,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%
2,3,1,1,1,IPv4,type 10.10.10.10,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%
3,0,NULL,NULL,NULL,IPv4,type Timeout,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%
4,0,NULL,NULL,NULL,IPv4,type Timeout,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%
5,3,1,1,1,IPv4,type 10.10.10.10,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%
6,3,1,1,1,IPv4,type 10.10.10.10,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%
7,3,1,1,1,IPv4,type 10.10.10.10,%%TRACERT: IPTYPE=IPv4, LOCALIP4="10.10.10.10", PEERIP4="10.10.10.10", MAXHOP=15;%%

To use it with find, you could do something like:

find /home/folder/file/source -iname "Tracert*" -execdir awk '
      /%%TRACERT:/ {sfx = $0; outfile = "filter.result." substr(FILENAME,3); next} 
      /IPv4 type/ {print $1","$2","$3","$4","$5","$6","$7" "$8","sfx > outfile}
' {} +

which will place the output files in the same directories where the input files are found. If the files are actually in a single directory, it would be simpler to cd there and then use the "local" awk command.

thank you sir, for find can you explain what substr(FILENAME,3) and {} + for? — ddrackochan, Oct 16 '19 at 17:52
@ddrackochan substr(FILENAME,3) just strips off the ./ relative path that is prepended to the filename by -execdir (otherwise the output files would look like filter.result../Tracert_1.txt). For the {} + syntax please refer to man find — steeldriver, Oct 16 '19 at 17:58

grep multi text file input and create diferent output each file

1 Answers1