Shell script: \n related problem to split a straightforward line

Question

That's an shell script snippet:

KVS_VARIABLES=$(awk -F= '!($1 && $2 && NF==2) { print "File failed validation on line " NR | "cat 1>&2"; next } { print $1, $2 }' $ENV_FILE_LOCATION)
echo ${KVS_VARIABLES}

for kv in ${KVS_VARIABLES}
do
  echo $kv
  key=$(echo $kv | awk -FS=" " '{print $1}')
  value=$(echo $kv | awk -FS=" " '{print $2}')

  echo "key: $key | value: $value"
done

I expect an output like:

key: VAR1 | value: VAL1
...

However, I'm getting that:

VAR1 VAL1 VAR2 VAL2 VAR3 VAL3
VAR1
key: VAR1 | value: 
VAL1
key: VAL1 | value: 
VAR2
key: VAR2 | value: 
VAL2
key: VAL2 | value: 
VAR3
key: VAR3 | value: 
VAL3
key: VAL3 | value:

EDIT

$ echo "$KVS_VARIABLES" ->
VAR1 VAL1
VAR2 VAL2
VAR3 VAL3

if you use double quotes like this.. then are you getting the one key value in each row ? echo "${KVS_VARIABLES}" — Kamaraj, Jun 28 '18 at 07:58
paste the output of echo "${KVS_VARIABLES}" ---> (use double quotes) — Kamaraj, Jun 28 '18 at 07:59
A simpler way to print to stderr from awk: print "some string" > "/dev/stderr" — glenn jackman, Jun 28 '18 at 11:43
You'll want to read http://mywiki.wooledge.org/BashFAQ/001: "How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?" The first advice is "Don't try to use "for". Use a while loop and the read command. " — glenn jackman, Jun 28 '18 at 11:45
Also read https://unix.stackexchange.com/q/171346/4667 about the importance of quoting variables. — glenn jackman, Jun 28 '18 at 11:46

score 1 · Answer 1 · edited Jun 28 '18 at 08:39

You need to specify IFS as newline, by default input file separator (IFS) includes a space and hence your for loop is behaving this way

KVS_VARIABLES=$(awk -F= '!($1 && $2 && NF==2) { print "File failed validation on line " NR | "cat 1>&2"; next } { print $1, $2 }' $ENV_FILE_LOCATION)
echo ${KVS_VARIABLES}
IFS=$(echo -en "\n\b")
for kv in ${KVS_VARIABLES}
do
  echo $kv
  key=$(echo $kv | awk  '{print $1}')
  value=$(echo $kv | awk  '{print $2}')    
  echo "key: $key | value: $value"
done

score 1 · Answer 2 · answered Jun 28 '18 at 08:40

Try:

#!/bin/bash

KVS_VARIABLES=$(echo -e "VAR1 VAL1\nVAR2 VAL2\nVAR3 VAL3\n")
echo "KVS_VARIABLES=[${KVS_VARIABLES}]"

while read -r kv; do
        echo "kv = ${kv}"
        key=$(echo $kv | awk '{print $1}')
        value=$(echo $kv | awk '{print $2}')

        echo "key: ${key} | value: ${value}"
done <<< "${KVS_VARIABLES}"

You should get

KVS_VARIABLES=[VAR1 VAL1
VAR2 VAL2
VAR3 VAL3]
kv = VAR1 VAL1
key: VAR1 | value: VAL1
kv = VAR2 VAL2
key: VAR2 | value: VAL2
kv = VAR3 VAL3
key: VAR3 | value: VAL3

score 0 · Answer 3 · answered Jun 28 '18 at 08:51

As Arushix points out, one of the issues with the code is that read splits the input on the values in the shell variable IFS (space, tab and newline by default).

However, I can't help but notice that your are parsing the variables in the original file several times; first by the first awk script, and then again with the read, and finally a third time with echo+awk. Each of these steps may throw its own set of errors.

If you just want to validate the variable assignments and print out the key/value pairs:

if ! awk -F '=' 'NF != 2 { exit 1 } { printf("key: %s | value: %s\n", $1, $2) }' "$conffile"; then
    echo 'Failed' >&2
    exit 1
fi

Given the input file

VAR1=VAL1
VAR2=VAL2
VAR3=VAL3
VAR4=VAL4

This will print

key: VAR1 | value: VAL1
key: VAR2 | value: VAL2
key: VAR3 | value: VAL3
key: VAR4 | value: VAL4

With an extra line reading just BOO at the end of the input file, the output would instead be

key: VAR1 | value: VAL1
key: VAR2 | value: VAL2
key: VAR3 | value: VAL3
key: VAR4 | value: VAL4
Failed

In addition to NF == 2, you may want to include a check for valid variable names etc.:

if ! awk -F '=' 'NF != 2 || $1 !~ /^[A-Za-z][A-Za-z0-9]+$/ { exit 1 } { printf("key: %s | value: %s\n", $1, $2) }' "$conffile"; then
    echo 'Failed' >&2
    exit 1
fi

Shell script: \n related problem to split a straightforward line

3 Answers3