looping through a file of two columns

Question

My question is basically a follow up question on this topic. I have a file like this:

1000 | line1
100  | line2
10   | line3

I want to do something on $2 if $1 is greater than 20. I wrote something to mimic the 2nd answer but it doesn't work:

for a, b in $(cat file.text|cut -d"|" -f 1,2);
do
if ($1>20) echo $2
done;

How can I achieve this? Thanks!

Have a look at How can I read a file (data stream, variable) line-by-line (and/or field-by-field)? -- there are example there about reading multiple fields from each line into variables. — glenn jackman, May 04 '18 at 17:47
You'll also want to read about Conditional blocks at http://mywiki.wooledge.org/BashGuide/TestsAndConditionals — glenn jackman, May 04 '18 at 17:49

DopeGhoti · Answer 1 · 2018-05-04T18:22:10.883

2

With awk:

$ for param in $(awk '$1 > 20 { print $3 }' inputfile); do ./process.sh "$param"; done

edited May 04 '18 at 18:22

answered May 04 '18 at 17:51

DopeGhoti

76,081

oh sorry I didn't say it clearly in the question - I am doing something other than just echo. I actually need to do ./process.sh $2. Right now my solution is use awk to generate a new script that repeats ./process.sh $2 many times. – user3768495 May 04 '18 at 18:16
I have modified my answer accordingly. It effectively does what you describe, but without actually writing a script. – DopeGhoti May 04 '18 at 18:22
Careful if the second column contains spaces... – Kusalananda May 04 '18 at 18:23
The second column as described is |. – DopeGhoti May 04 '18 at 18:24
Thanks @DopeGhoti! This is a great answer as well. Too bad system doesn't allow me accept two answers. – user3768495 May 04 '18 at 18:52
@DopeGhoti Sorry, I meant the 3rd column, the data after the pipe. – Kusalananda May 04 '18 at 21:10

Kusalananda · Answer 2 · 2018-05-04T18:22:44.050

You wouldn't really want to parse data in the shell itself, but to use a tool like awk to do it for you.

awk -F '[[:blank:]|]+' '$1 > 20 { print $2 }' file

This instructs awk to treat each line of the file as a set of fields delimited by any number of pipe-symbols or blanks (spaces or tabs). When the first field is greater than 20, the second field is printed.

Why is using a shell loop to process text considered bad practice?

If the second column contain blanks, then you may want to use [[:blank:]]*[|][[:blank:]]* as the delimiter:

awk -F '[[:blank:]]*[|][[:blank:]]*' '$1 > 20 { print $2 }' file

The [|] matches a literal pipe symbol, and may be replaced by \\| (if you enjoy backslashes).

The following executes a script with each instance of the second column whose first column is greater than 20 as the command line argument:

awk -F '[[:blank:]]*[|][[:blank:]]*' '$1 > 20 { print $2 }' file |
xargs -I XX ./script.sh XX

With something like -P 4 as an option to xargs, you can run multiple parallel instances of your script at once (four with -P 4).

score 1 · Accepted Answer · answered May 04 '18 at 18:03

awk -F'|' '$1 > 20 { system("/path/to/another/script.sh "$2 }' < file.text

This tells awk to split the input up into fields based on the pipe symbol. Any first field whose value is greater than 20 triggers the system call to ... anything you want. Keep in mind that the argument (here $2, but it could be $0 or any other calculation you do in awk) is passed to the script via a shell call, so if those values can contain shell-special characters, carefully quote it. I'll refer to one of Stéphane's answers for an example of how to do that:

awk 'function escape(s) {
       gsub(/'\''/,"&\\\\&&",s)
       return "'\''" s "'\''"
     }
     { system("/path/to/another/script.sh" escape($2)) }'

Yes this is exactly what I am looking for. I used a work-around like this:awk -F"|" '{if ($1>20) print "script.sh " $2}' |bash. But your solution is better! Thanks! — user3768495, May 04 '18 at 18:17

score 0 · Answer 4 · answered May 04 '18 at 20:37

0

You can try with gnu sed too

sed -E 's/([0-9]*) \| (.*)/[ \1 -gt 20 ] \&\& echo do something with \2/e' infile

answered May 04 '18 at 20:37

ctac_

1,960

looping through a file of two columns

4 Answers4