13

I'm trying to set an RSA key as an environment variable which, as a text file, contains newline characters.

Whenever I attempt to read from the file and pass it into an environment variable, it will just stop at the newline character on the first line. How can I prevent this?

notfreshprince
  • 133
  • 2
    please provide example of: 1) text file, 2)how do you read from the file, 3) how do you pass it to an env. variable 4)the content of the variable – Yaron Jun 08 '17 at 12:59

4 Answers4

9

Note that except in zsh, shell variables cannot store arbitrary sequences of bytes. Variables in all other shells can't contain the NUL byte. And with the yash, they can't contain bytes not forming valid characters.

For files that don't contain NUL bytes, in POSIX-like shells, you can do:

var=$(cat file; echo .); var=${var%.}

We add a .\n and strip the trailing . to work around the fact that $(...) strips all trailing newline characters.

The above would also work in zsh for files that contain NULs though in zsh you could also use the $mapfile special associative array:

zmodload zsh/mapfile
var=$mapfile[file]

In zsh or bash, you can also use:

{ IFS= read -rd '' var || :; } < file

That reads up to the first NUL byte. It will return a non-zero exit status unless a NUL byte is found. We use the command group here to be able to at least tell the errors when opening the file, but we won't be able to detect read errors via the exit status.

Remember to quote that variable when passed to other commands. Newline is in the default value of $IFS, so would cause the variable content to be split when left unquoted in list contexts in POSIX-like shells other than zsh (not to mention the other problems with other characters of $IFS or wildcards).

So:

printf %s "$var"

for instance (not printf %s $var, certainly not echo $var which would add echo's problems in addition to the split+glob ones).

With non-POSIX shells:

Bourne shell:

The bourne shell did not support the $(...) form nor the ${var%pattern} operator, so it can be quite hard to achieve there. One approach is to use eval and quoting:

eval "var='`printf \\' | cat file - | awk -v RS=\\' -v ORS= -v b='\\\\' '
  NR > 1 {print RS b RS RS}; {print}; END {print RS}'`"

With (t)csh, it's even worse, see there.

With rc, you can use the ``(separator){...} form of command substitution with an empty separator list:

var = ``(){cat file}
Stéphane Chazelas
  • 544,893
4

One way to do this (including the case of 0x0) is by storing any binary data in an encoded form in shell variables.

One can use the Base64 Encoding (see RFC 4648) for this.

For example:

encoded_binary="$( cat binary | base64 -w 0 )"

The -w 0 is rather cosmetic (it saves a tiny bit of memory).

When the binary data is to be used, this can only be done by piping or redirecting it, after it has been decoded again.

For example:

{
        printf '%s\n' "${passphrase}"
        printf '%s' "${encoded_binary}" | base64 -d
} | \
gpg --quiet --batch --passphrase-fd 0 --output - --decrypt

In this example, the binary would have been a OpenPGP message that is to be decrypted with gpg.

Instead of base64, uuencode/uudecode (which are specified by POSIX) may be used.

encoded_binary="$( uuencode -m '/dev/stdout' )"
...
printf '%s' "${encoded_binary}" | uudecode -o '/dev/stdout'

But beware, that /dev/stdout is ought to be a special symbol (and not a file). Some implementations of uudecode support this properly (e.g. GNU), while others don't (e.g. BusyBox' which only supports it since it’s commit c7b90dc4d10ccc4f95940f42676ff907cee73272 and only if some special build option is set).

Also, if you have constructs like:

encoded_binary="$(read_function | base64 -w 0)"

You may still want to check the exit status of read_function (or the same when decoding), which is not directly possible in the POSIX Shell Command Language, but indrectly with constructs like this.

calestyo
  • 147
0

If you use zsh, you can just assign the contents of a file like this:

p="`cat file_to_be_read`"
saga
  • 1,401
  • 4
    Quoting here is not necessary (though won't harm). p=\cat file_to_be_read`` or the p=$(cat file_to_be_read) modern equivalent will store the content of the file without the trailing newline characters (and provided the file doesn't contain null bytes in shells other than zsh) just the same. It's important to quote $p when it's used later on though (in shells other than zsh/fish/rc) using "$p" in Bourne-like shells and $p:q in csh-like shells. – Stéphane Chazelas Jun 08 '17 at 13:24
  • 1
    You can also skip the cat and instead use p=$(< file_to_be_read) – thrig Jun 08 '17 at 13:46
0

On traditional bourne shells which don't support the parameter manipulation schemes, we might do the following:

NL='
'
var=`cat file`
while case `printf '%s\n' "$var" | wc -l` in "`wc -l < file`" ) break ;; esac
do var=$var$NL; done

Wherein we first start with the contents of the file named file. Now if there are any trailing newlines then only the while loop kick in. The non-trailing newlines will not any issues since they aren't touched by the command substitution.

On every iteration of the while loop, a newline is pegged to the variable, and since this doesn't involve command sub, aka, ..., no loss of newlines occurs.