Replace environment variables in a file with their actual values?

Question

Is there an easy way to substitute/evaluate environment variables in a file?

Like let's say I have a file config.xml that contains:

<property>
    <name>instanceId</name>
    <value>$INSTANCE_ID</value>
</property>
<property>
    <name>rootPath</name>
    <value>/services/$SERVICE_NAME</value>
</property>

...etc.

I want to replace $INSTANCE_ID in the file with the value of the INSTANCE_ID environment variable, $SERVICE_NAME with the value of the SERVICE_NAME env var.

I won't know a priori which environment vars are needed (or rather, I don't want to have to update the script if someone adds a new environment variable to the config file).

Answer 1

You could use envsubst (part of gnu gettext):

envsubst < infile

will replace the environment variables in your file with their corresponding value. The variable names must consist solely of alphanumeric or underscore ASCII characters, not start with a digit and be nonempty; otherwise such a variable reference is ignored.

Some alternatives to gettext envsubst that support ${VAR:-default} and extra features:
rust alternative
go alternative
node.js alternative

---

To replace only certain environment variables, see this question.

Answer 2

This is not very nice but it works

( echo "cat <<EOF" ; cat config.xml ; echo EOF ) | sh

If it was in a shell script it would look like:

#! /bin/sh
cat <<EOF
<property>
    <name>instanceId</name>
    <value>$INSTANCE_ID</value>
</property>
EOF

Edit, second proposal:

eval "echo \"$(cat config.xml)\""

Edit, not strictly related to question, but in case of variables read from file:

(. .env && eval "echo \"$(cat config.xml)\"")

Answer 3

If you happen to have Perl (but not gettext and envsubst) you can do the simple replacement with a short script:

$ export INSTANCE_ID=foo; export SERVICE_NAME=bar;
$ perl -pe 's/\$([_A-Z]+)/$ENV{$1}/g'  < config.xml
<property>
    <name>instanceId</name>
    <value>foo</value>
</property>
<property>
    <name>rootPath</name>
    <value>/services/bar</value>
</property>

I assumed the variable names will only have uppercase letters and underscores, but the first pattern should be easy to alter as needed. $ENV{...} references the environment Perl sees.

If you want to support the ${...} syntax or throw an error on unset variables, you'll need some more work. A close equivalent of gettext's envsubst would be:

perl -pe 's/\$(\{)?([a-zA-Z_]\w*)(?(1)\})/$ENV{$2}/g'

Though I feel that feeding variables like that via the process environment seems a bit iffy in general: you can't use arbitrary variables in the files (since they may have special meanings), and some of the values could possibly have at least semi-sensitive data in them.

Answer 4

The command below replaces all occurrences of environment variables of the form $VAR in a file.

compgen -e | xargs -I @ sh -c 'printf "s|\$%q\>|%q|g\n" "@" "$@"' | sed -f /dev/stdin input.file > output.file

Note: sed syntax is slightly different on macOS: [[:>:]] to be used instead of \>.

Here's how it works:

• compgen -e lists environment variable names without values, for example:

  HOME
  LANG
  PATH
  PWD
  ...
  

• xargs redirects the output to the shell that renders the sed substitute commands via printf populating the environment variable values along the way:

  s|$HOME\>|/Users/sshymko|g
  s|$LANG\>|en_US.UTF-8|g
  s|$PATH\>|/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin|g
  s|$PWD\>|/Users/sshymko/Documents|g
  ...
  

• sed modifies contents from a file using the substitute commands in the standard input

The substitute command format is s|\$%q\>|%q|g\n, where:

• | — delimiter guaranteed to be escaped in environment variable name/value
• %q — string placeholder of printf escaping special shell characters
• \> — end of a word boundary to avoid a variable name prefix match
• g — global replace, i.e. replace all occurrences
• \n — line break separating sed substitute commands

Additionally:

• @ — argument placeholder of xargs holding the environment variable name
• $@ — dereferenced value of the environment variable

Answer 5

Continuing with the Perl and PHP examples, here is a Python oneliner:

python -c 'import os,sys; sys.stdout.write(os.path.expandvars(sys.stdin.read()))' < in.file > out.file

Cause not every environment has envsubst installed, unfortunately.

Answer 6

With zsh:

zmodload zsh/mapfile
set -o extendedglob
mapfile[config.xml]=${mapfile[config.xml]//(#m)\$[[:IDENT:]]##/${(e)MATCH}}

To also cover ${var}:

mapfile[config.xml]=${mapfile[config.xml]//(#m)\$([[:IDENT:]]##|{[[:IDENT:]]##})/${(e)MATCH}}

To cover all expansions, including `cmd`, $(( 1+1 )), $(cmd), $var[x], ${var#???}...:

mapfile[config.xml]=${(e)mapfile[config.xml]}

Answer 7

May I suggest my own script for this?

https://github.com/rydnr/set-square/blob/master/.templates/common-files/process-file.sh

#!/bin/bash /usr/local/bin/dry-wit
# Copyright 2016-today Automated Computing Machinery S.L.
# Distributed under the terms of the GNU General Public License v3

function usage() {
cat <<EOF
$SCRIPT_NAME -o|--output output input
$SCRIPT_NAME [-h|--help]
(c) 2016-today Automated Computing Machinery S.L.
    Distributed under the terms of the GNU General Public License v3

Processes a file, replacing any placeholders with the contents of the
environment variables, and stores the result in the specified output file.

Where:
    * input: the input file.
    * output: the output file.
Common flags:
    * -h | --help: Display this message.
    * -v: Increase the verbosity.
    * -vv: Increase the verbosity further.
    * -q | --quiet: Be silent.
EOF
}

# Requirements
function checkRequirements() {
  checkReq envsubst ENVSUBST_NOT_INSTALLED;
}

# Error messages
function defineErrors() {
  export INVALID_OPTION="Unrecognized option";
  export ENVSUBST_NOT_INSTALLED="envsubst is not installed";
  export NO_INPUT_FILE_SPECIFIED="The input file is mandatory";
  export NO_OUTPUT_FILE_SPECIFIED="The output file is mandatory";

  ERROR_MESSAGES=(\
    INVALID_OPTION \
    ENVSUBST_NOT_INSTALLED \
    NO_INPUT_FILE_SPECIFIED \
    NO_OUTPUT_FILE_SPECIFIED \
  );

  export ERROR_MESSAGES;
}

## Parses the input
## dry-wit hook
function parseInput() {

  local _flags=$(extractFlags $@);
  local _flagCount;
  local _currentCount;

  # Flags
  for _flag in ${_flags}; do
    _flagCount=$((_flagCount+1));
    case ${_flag} in
      -h | --help | -v | -vv | -q)
         shift;
         ;;
      -o | --output)
         shift;
         OUTPUT_FILE="${1}";
         shift;
         ;;
    esac
  done

  # Parameters
  if [[ -z ${INPUT_FILE} ]]; then
    INPUT_FILE="$1";
    shift;
  fi
}

## Checking input
## dry-wit hook
function checkInput() {

  local _flags=$(extractFlags $@);
  local _flagCount;
  local _currentCount;
  logDebug -n "Checking input";

  # Flags
  for _flag in ${_flags}; do
    _flagCount=$((_flagCount+1));
    case ${_flag} in
      -h | --help | -v | -vv | -q | --quiet)
         ;;
      -o | --output)
         ;;
      *) logDebugResult FAILURE "fail";
         exitWithErrorCode INVALID_OPTION ${_flag};
         ;;
    esac
  done

  if [[ -z ${INPUT_FILE} ]]; then
    logDebugResult FAILURE "fail";
    exitWithErrorCode NO_INPUT_FILE_SPECIFIED;
  fi

  if [[ -z ${OUTPUT_FILE} ]]; then
      logDebugResult FAILURE "fail";
      exitWithErrorCode NO_OUTPUT_FILE_SPECIFIED;
  fi
}

## Replaces any placeholders in given file.
## -> 1: The file to process.
## -> 2: The output file.
## <- 0 if the file is processed, 1 otherwise.
## <- RESULT: the path of the processed file.
function replace_placeholders() {
  local _file="${1}";
  local _output="${2}";
  local _rescode;
  local _env="$(IFS=" \t" env | awk -F'=' '{printf("%s=\"%s\" ", $1, $2);}')";
  local _envsubstDecl=$(echo -n "'"; IFS=" \t" env | cut -d'=' -f 1 | awk '{printf("${%s} ", $0);}'; echo -n "'";);

  echo "${_env} envsubst ${_envsubstDecl} < ${_file} > ${_output}" | sh;
  _rescode=$?;
  export RESULT="${_output}";
  return ${_rescode};
}

## Main logic
## dry-wit hook
function main() {
  replace_placeholders "${INPUT_FILE}" "${OUTPUT_FILE}";
}
# vim: syntax=sh ts=2 sw=2 sts=4 sr noet

Answer 8

Use cmake's configure_file function.

It copies an <input> file to an <output> file and substitutes variable values referenced as @VAR@ or ${VAR} in the input file content. Each variable reference will be replaced with the current value of the variable, or the empty string if the variable is not defined.

Answer 9

To add to the discussion about Docker images above (unfortunately couldn't comment there).

As the safest methods are to use either envsubst or perl. When used in Docker, the problem is that the latter is a part of Debian-based images (including slim-type) but not Alpine. Adding envsubst to an image directly, will result in increasing the size for 25MB or 65MB in Alpine or Debian-based images correspondingly.

As the result, I ended up with selection of either one in docker-entrypoint directly. A caveat: in Alpine-based images, it'll download envsubst directly from Github. You might want it to have it somewhere locally to speed up the process. Or bake-in during the build stage (just 2.5Mb that way).

        # do not add `envsubst` to the image! It saves 25/65M in Docker Alpine/Debian-based images correspondingly
        # `perl` is part of Debian but not Alpine-based images
        if [[ -n $(command -v perl) ]] ; then
            perl -pe 's/\$(\w+)/$ENV{$1}/g' <"${file}" >"/tmp/${filename}"
        else
            if [[ ! -f /tmp/envsubst ]] ; then
                wget -O /tmp/envsubst "https://github.com/a8m/envsubst/releases/latest/download/envsubst-$(uname -s)-$(uname -m)"
                chmod 500 /tmp/envsubst
            fi
            /tmp/envsubst -i "${file}" -o "/tmp/${filename}"
        fi
...
[[ -f /tmp/envsubst ]] && rm -f /tmp/envsubst

Answer 10

Similarly to the Perl answer, environment variable substitution can be delegated to the PHP CLI. Dependency on PHP may or may not be acceptable depending on the tech stack in use.

php -r 'echo preg_replace_callback("/\\$([a-z0-9_]+)/i", function ($matches) { return getenv($matches[1]); }, fread(STDIN, 8192));' < input.file > output.file

You can go further and put it in a reusable script, for example, envsubst:

#!/usr/bin/env php
<?php

echo preg_replace_callback(
    '/\$(?<name>[a-z0-9_]+)/i',
    function ($matches) {
        return getenv($matches['name']);
    },
    file_get_contents('php://stdin')
);

The usage would be:

envsubst < input.file > output.file

Answer 11

Building upon @don_crissti's answer, which is definitely correct, but it doesn't address a last mile concern in the question, which is:

I want to replace $INSTANCE_ID in the file with the value of the INSTANCE_ID

Interpolating a file, "in-place", using envsubst, is actually trickier than it would appear, given that read and write operations are (mostly) done concurrently, and are often parallel, when more than one cpu is available.

Eenvsubst reads against STDIN, and writes against STDOUT, we can't simply read, interpolate and redirect stdout to the same template file:

/ # </tmp/tmpl cat
hello $PATH
/ # </tmp/tmpl envsubst
hello /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/ # </tmp/tmpl envsubst >/tmp/tmpl 
/ # </tmp/tmpl cat
/ # stat /tmp/tmpl 
  File: /tmp/tmpl
  Size: 0           Blocks: 0          IO Block: 4096   regular empty file

There are a lot of ways to approach this, but the all boil down to: write the interpolated result to a buffer or write to a different file. Because your example looks config and deploy related, I prefer to use a makeesque build and install workflow, where we write the interpolated results to a dist or build directory, and then work against that.

# copy configuration/manifest/etc to dist directory, and then interpolate tmpl; not tested at all 
$ cp -rf path/to/configs ./dist
$ find -L path/to/configs -type f -print0 \
    | xargs -0 -n1 -- sh -c '<"$$1" envsubst >"dist/$$1"' _ 

Cheers from Linkoping ;)

Answer 12

Another way without envsubst could be:

printenv | sed 's|=.*||' | while read envvarname; do sed -i "s|\${$envvarname}|${!envvarname}|" /file/to/replace.txt; done

Will replace the string ${MYVAR} with the content of MYVAR in the file /file/to/replace.txt.

Answer 13

When using envsubst and the environment variable does not exist, envsubst puts an empty string to the output which may be a problem. The below script checks that all variables exist before running envsubst.

FILE_TEMP="file_with_env_declarations.txt"
missing_vars=false
while read line; do
  in_env=$(env | grep $line)
  [ "${in_env}" = "" ] && echo "ERROR: The variable '$line' does not exist!" && missing_vars=true
done < <(cat ${FILE_TEMP} | egrep -o "\${[A-Za-z0-9_]+}|\$[A-Za-z0-9_]+" | tr -d '${}')
[ "${missing_vars}" = true ] && echo "ERROR: There are variables in '${FILE_TEMP}' missing in the environment!" && exit 1
envsubst < ${FILE_TEMP}