1

Edit -- Not using bash $1 variables. Example is using field designators: $1, $2, $3 are the awk input record fields Unfortunately I did the example too quickly and mixed-up the example.

I believe this is a generic problem. I am using awk as the demonstration tool. The aim is to insinuate an (awk) script parameter into the awk (or any tool) command.

We are using fields from STDIN to as provided withn awk to produce the a result. The identifiers $1, $2, $3 are the awk fields from the STDIN stream.

Desired solution A (prefered concept), something like...

#!/bin/bash  -f
#   script: blue.sh
#
    awk -e '{ print $1 " and " $2 " silly example" ;}'
#

And the desired output:

$  echo  "red blue yellow" > ./blue.sh 
red and blue silly example
$

Alternaitve B can be thought of perhaps:

#!/bin/bash  -f
#   script: green.sh
#
    wrk="awk -e '{ print \$3 \" and \" \$2 \" variable example\" ;}'  "
#
    echo "work is:"
    echo $wrk
    echo
    $wrk

And the desired output:

$  echo  "red blue yellow" > ./green.sh 
wrk is:
awk -e '{ print $3 " and " $2 " variable example" ;}'  

yellow and blue variable example
$

The problem comes in the quoting for the -e expression.

Is there a means to achieve either or these examples?

will
  • 500

2 Answers2

4

This is probably a duplicate of "How can we run a command stored in a variable?", but I'll answer it for your particular case since it has multiple parts to it:

  1. Storing a command in a variable and running it using the variable.
  2. Passing data into awk.

The "run command in variable" bit

Here's a suggestion for an alternative implementation of your script:

#!/bin/bash

wrk=( awk -v a="$3" -v b="$2"
      'BEGIN { printf "%s and %s variable example\n", a, b }' )

awk -v a="$1" -v b="$2" 'BEGIN { printf "%s and %s silly example\n", a, b }'

printf 'wrk is: %s\n' "${wrk[*]}"
"${wrk[@]}"

Running this:

$ ./blue.sh "red" "blue" "yellow"
red and blue silly example
wrk is: awk -v a=yellow -v b=blue BEGIN { printf "%s and %s variable example\n", a, b }
yellow and blue variable example

What I do is that I store the awk command in an array instead of in a string. This ensures that each separate element of the array may be properly quoted while still remaining a separate element, separate from the other parts of the array. For example, the awk word at the start is one word separate from the last entry in the array which is the entire awk code.

When using "${wrk[@]}" as a command, each element of the array is used as individual command line arguments (the first element being the command). The quoting of the "${wrk[@]}" expansion ensures that each element is individually quoted (using ${wrk[@]} would not work).

That's how you store and run a command in a variable, i.e. you use an array and make sure to properly quote the array elements and the expansion of the array.

For the printf call in the shell script, I use "${wrk[*]}" to generate a single string from the wrk array (all elements separated by spaces by default). You will notice that this does not preserve any quoting of the individual elements, which is to be expected since the shell removed these when creating the array (just like the quotes are removed when doing things like a='hello world').

To see each separate element of wrk, use something like printf 'element: "%s"\n' "${wrk[@]}" instead. This would output

element: "awk"
element: "-v"
element: "a=yellow"
element: "-v"
element: "b=blue"
element: "BEGIN { printf "%s and %s variable example\n", a, b }"

In this particular case, and depending on your needs, you may also be served by using a shell function:

#!/bin/bash

wrk () {
    awk -v a="$1" -v b="$2" 'BEGIN { printf "%s and %s variable example\n", a, b }'
}


awk -v a="$1" -v b="$2" 'BEGIN { printf "%s and %s silly example\n", a, b }'

wrk "$3" "$2"

Note that $1 and $2 in the function refers to the function's first and second argument, not the script's.

Running this:

$ ./blue.sh "red" "blue" "yellow"
red and blue silly example
yellow and blue variable example

The awk-specific bit

To pass data into awk I'm using -v to initialise awk variables on the awk command line. This is preferable to injecting variables into the awk code via shell expansions since doing that would potentially allow a user to inject executable code rather than just data (by simply making $3 contain a ; and some awk code, for example). See "Command line argument in awk" and similar question on this site for more about that.

You may also pass data into awk using environment variables:

a=$3 b=$2 awk 'BEGIN { printf "%s and %s variable example\n", ENVIRON["a"], ENVIRON["b"] }' )

or,

export a="$3"
export b="$2"
awk 'BEGIN { printf "%s and %s variable example\n", ENVIRON["a"], ENVIRON["b"] }' )

I'm also executing the awk code in a single BEGIN block since there is no input to read on standard input.

Kusalananda
  • 333,661
  • That's afeasible approach where you have bash variables a and b. The goal fo rme is to be processing lines from STDIN through a what is effectively a template of: $3 " and " $2 " variable example". Use of aBEGIN{ }closure is a different use case, also at I need to update the question, the$1,$2,$3variables are theawk` input fields mea culpa on for that. – will Mar 05 '20 at 21:50
0

I have found a solution, though it turned-out to be quite different to the initial vision. I was unable to find a way to expand bash-script variables in a command so that has to be 'constructed'. It turned-out that xargswas unsuitanble for several reasons to do with how the shell wanted to do its thing.

I will present an example run and then the code and describe how/what is happening. I have marked areas to talk on with a label such as "# (2)".

Example:

input ...

$ svn status -qu ..
M            28637   /src/line.h
M            28637   /src/line.cpp
        *    28637   /testing/CheckWindows.py

processing ...

The repeat script takes two parameters:

  1. A template or one or more commands to execute on the list of inputs
  2. Some awk commands to filter the input stream

command line:

 svn status -qu .. | repeat                                        \
                       'cp -v $ff  /mnt/testhost/testarea/r02/$ff' \
                       "{  if( $1 == "M" ){ print  $3; } }"

output ...

'/src/line.h' -> '/mnt/testhost/testarea/r02/src/line.h'
'/src/line.cpp' -> '/mnt/testhost/testarea/r02/src/line.cpp'

What happened?

  • Only the locally modified files are copied to the r02/testarea
    • Clearly the filtering can be anything
  • The purpose of the repeat is to repeat a set of command(s) on a filtered list from input

Annotated code:

    #!/bin/bash
    #   repeat.bash

    function  repeat
    {
        local -a    actions=()
        local       DELIMITER=";"
        local       filter="${2}"
        local       cmd=""

        local       wrk="${1}${DELIMITER}"        # (1)   Template
        local       str=""


        while [[ $wrk  ]]                         # (2)   Split-up individual commands
        do
            str="${wrk%%"${DELIMITER}"*}"
            actions+=( "${str}" )

            wrk="${wrk#*"${DELIMITER}"}"
        done

        echo    "actions: "                       # (3)   Debugging code

        for action in "${actions[@]}"
        do
            echo  "        => \"${action}\"."
        done
        echo  "filter  => '${filter}'."


        ex="${filter}"                            # (4)   Awk instructions to 
                                                  #       process input stream


        for ff in $(awk -e "${ex}")               # (5)   .
        do
            for action in "${actions[@]}"         # (6)   .
            do
                cmd="${action//\$\{ff\}/${ff}}"   # (7)   .
                cmd="${cmd//\$ff/${ff}}"

                ${cmd}                            # (8)   .
            done
        done

        return

    }

    repeat  "${@:1}"                              # (9)   .

Notes:

  1. Arg $1 is the command template.
    *
    An extra delimiter is added to the end for later parsing
  2. $1 is usually more than one command. Split the input argument into individual semi-colon separated command strings.
  3. Print the parameters for confirmation.
  4. Load the filter into the ex variable.
    *
    In this example ex is only the filter passed as $2.
  5. Loop over each string generated from the filtered input line.
    *
    ex becomes the awk commands to be executed as the selection/generate filter.
  6. Loop over all the action command templates.
  7. Expand $ff as well as ${ff} syntax within the action command string.
  8. Execute each generated command line for each file: $ff.
  9. Call the function to execute the script.

As it turned-pout this became quite different because it wasn't directly possible to inject arguments into the awk or template strings. However the work-arounds are not too inconvenient and show how it all works without cryptic expansion and so on.

I trust the idea is useful. I find it very handy now that I have a working snippet.

will
  • 500