Read non-bash variables from a file into bash script

Question

My bash script needs to read the variables from a text file that contains hundreds of variables, most of which conform to standard bash variable syntax (e.g., VAR1=VALUE1). However, a few lines in this file can be problematic, and I hope to find a simple solution to reading them into bash.

Here's what the file looks like:

#comment
VAR1=VALUE1
VAR_2=VALUE_2
...
VAR5=VALUE 5 (ASDF)
VAR6=VALUE 6 #a comment
VAR7=/path/with/slashes/and,commas
VAR8=2.1
VAR9=a name with spaces
VAR_10=true
...
VAR_N=VALUE_N

The rules about the file structure include:

• one variable (with value) per line
• the assignment (=) has no spaces around it
• the variable name is in the first column (unless it is a comment line)
• comments can follow the value (after #)
• the values can include spaces, parens, slashes, commas, and other chars
• the values can be floating point numbers (2.1), integers, true/false, or strings.
• string values are not quoted, and they can be a thousand chars long or longer
• the variable name contains only letters and underscores UPDATE: and numbers.

Most of the variables are of a type that would just allow me to source the file into my bash script. But those few problematic ones dictate a different solution. I'm not sure how to read them.

Answer 1

While you can transform this file to be a shell snippet, it's tricky. You need to make sure that all shell special characters are properly quoted.

The easiest way to do that is to put single quotes around the value and replace single quotes inside the value by '\''. You can then put the result into a temporary file and source that file.

script=$(mktemp)
sed <"config-file" >"$script" \
  -e '/^[A-Z_a-z][A-Z_a-z]*=/ !d' \
  -e s/\'/\'\\\\\'\'/g \
  -e s/=/=\'/ -e s/\$/\'/

I recommend doing the parsing directly in the shell instead. The complexity of the code is about the same, but there are two major benefits: you avoid the need for a temporary file, and the risk that you accidentally got the quoting wrong and end up executing a part of a line as a shell snippet (something like dangerous='$(run me)'). You also get a better chance at validating potential errors.

while IFS= read -r line; do
  line=${line%%#*}  # strip comment (if any)
  case $line in
    *=*)
      var=${line%%=*}
      case $var in
        *[!A-Z_a-z]*)
          echo "Warning: invalid variable name $var ignored" >&2
          continue;;
      esac
      if eval '[ -n "${'$var'+1}" ]'; then
        echo "Warning: variable $var already set, redefinition ignored" >&2
        continue
      fi
      line=${line#*=}
      eval $var='"$line"'
  esac
done <"config-file"

Answer 2

Assuming your contents are in a file x.txt, you can use a sed expression:

sed -e 's/#.*//' -e 's/=.*$/="&"/g' -e 's/=//2' x.txt

First expression puts everything after the = sign in quotes. Second expression removes extra = sign and the third expression removes everything between # and " as they are comments.

Following will quote RHS into single quotes:

sed -e "s/#.*//" -e "s/=.*$/='&'/g" -e "s/=//2" x.txt