Passing script variables into AWK after pipe

Question

I'm having an issue getting variable assignment to work in a script I've written that uses awk. The awk command sits inside of a function and there are three variables that are passed to that function elsewhere in the script. I made this work previously by breaking the awk command with the variable: awk ... '"$var"' ... , but apparently this is a rather sloppy way of doing it, and awk allows variable assignemnt via the -v switch.

This is what I'm trying to do:

my_function() {
  command1 \
  | command2 \
  | awk -v var1="$1" -v var2="$2" -v var3="$3" '{
                FS="[ ,\"]+"}/var1=/{
                        for ( i=2; i<NF; i++) {
                                        if ( var2 ){
                                                var3; next;
                                        }
                                }
                        }
                {
                        fname=$0
        }'
}
...
my_function "$firstArg" "$secondArg" "$thirdArg"

Based on the output from using set -x, I can see that the variable assignment are made properly in the beginning of the awk command, but the awk command simply returns nothing as oppossed to doing it the awk ... '"$var"' ... way, which does work properly.

I feel like I'm just missing something here.

/var1=/ is a regexp constant - if you want to use a variable in a regex test, you'd need to do it like $0 ~ var1"=". See Pass shell variable as a /pattern/ to awk — steeldriver, Aug 17 '21 at 01:57
Regarding This is what I'm trying to do: - no, that's a block of code that doesn't do whatever it is you're trying to do. We can't tell what you're really trying to do by reading code that doesn't do what you want, you'd have to specifically tell us what you're trying to do and provide some sample input/output to demonstrate it for us to be able to fully help you. — Ed Morton, Aug 17 '21 at 15:54

Ed Morton · Accepted Answer · 2021-08-17T16:29:10.963

It seems like this might be what you're trying to do:

my_function() {
    command1 |
    command2 |
    awk -v var1="$1" -v var2="$2" -v var3="$3" '
        BEGIN { FS="[ ,\"]+" }
        $0 ~ (var1"=") {
            for ( i=2; i<NF; i++) {
                if ( $0 ~ var2 ) {   # maybe just "if (var2)", depends on context
                    var3        # NOTE: this line does nothing
                    next
                }
            }
        }
        { fname=$0 }      # NOTE: this is populating a variable thats never used.
    '
}
...
my_function "$firstArg" "$secondArg" "$thirdArg"

Yes, awk ... '"$var"' ... is horrible. Consider:

$ var=$(printf 'a\nb')
$ echo "$var"
a
b

Obvious, Desirable output:

$ awk -v foo="$var" 'BEGIN{print foo}'
a
b

Obscure, undesirable output:

$ awk 'BEGIN{print '"$var"'}'
$ awk 'BEGIN{print "'"$var"'"}'
awk: cmd. line:1: BEGIN{print "a
awk: cmd. line:1:             ^ unterminated string
awk: cmd. line:1: BEGIN{print "a
awk: cmd. line:1:             ^ syntax error
$

See how-do-i-use-shell-variables-in-an-awk-script for a full discussion on the different ways to use the values of shell variables in an awk script. Using -v might be the wrong approach depending on the contents of your shell variables and using ENVIRON[] or ARGV[] might be better.

I changed {FS="[ ,\"]+"} to BEGIN{FS="[ ,\"]+"} because the way you're doing it in the script in your question is setting it once per input line but AFTER the line has already been split so the FS you're setting won't be applied to the first input line and you're doing the work of setting it once per line instead of once total. Look at the difference:

$ printf 'a,b\nc,d\n' | awk '{FS=","} {print $1}'
a,b
c
$ printf 'a,b\nc,d\n' | awk 'BEGIN{FS=","} {print $1}'
a
c

It's not clear what you're hoping var3; will do but what it WILL do is - nothing.

Note how I've indented and used line wrapping so that the structure of the code reflects the control flow of your program. The indenting etc. in the code in your question doesn't do that and so obfuscates your code - try to structure your code following the style I show above in future and run it through gawk -o- for pretty-printing if you're not sure how to structure it.

Passing script variables into AWK after pipe

1 Answers1