How to print a variable that contains unprintable characters?

Question

I want to display the value of the $IFS variable, which could contain unprintable characters (for example: newline).

I used the following command to do so:

echo -n "$IFS" | hexdump -C

Which worked great in my case.

But is there anything wrong in using this command? For example, does echo replaces some unprintable characters with some other characters before printing them to its stdout, or some other problem like that?

Answer 1

Some various approaches at giving visual representations of strings:

POSIX

$ printf %s "$IFS" | od -vtc -to1
0000000      \t  \n  \0
        040 011 012 000
0000004
$ printf '%s\n' "$IFS" | LC_ALL=C sed -n l
 \t$
\000$

(the extra \n is necessary as otherwise the behaviour of sed is unspecified if the last line doesn't end in a newline). A POSIX sh won't have NUL in $IFS like my zsh does here. The behaviour of sed is unspecified if the input contains NULs.

Shell builtins

• typeset -p (ksh, zsh, bash, yash) might give you an unambiguous output for some strings.

  $ ksh93 -c 'typeset -p IFS'
  IFS=$' \t\n'
  $ zsh -c 'typeset -p IFS'
  typeset IFS=$' \t\n\C-@'
  $ mksh -c 'typeset -p IFS'
  typeset IFS=$' \t\n'
  $ a=$'\u00e9e\u301\u200b' ksh -c 'typeset -p a'
  typeset -x a=$'\u[e9]e\u[301]\u[200b]'
  

  but for that latter one (using Unicode combining acute accent and zero-width space characters), zsh/mksh are not helping (even with LC_ALL=C typeset -p a with mksh -o utf8-mode). bash's output is generally not unambiguous when sent to a terminal.

• printf %q with GNU printf and the printf builtin of ksh93, zsh and bash:

  $ a=$'\u00e9e\u301\u200b' bash -c 'printf "%q\n" "$IFS" "$a" ""'
  $' \t\n'
  éé​
  ''
  $ a=$'\u00e9e\u301\u200b' ksh -c 'printf "%q\n" "$IFS" "$a" ""'
  $' \t\n'
  $'\u[e9]e\u[301]\u[200b]'
  ''
  \ $'\t'$'\n'$'\0'
  éé​
  ''
  $ a=$'\u00e9e\u301\u200b' sh -c '/usr/bin/printf "%q\n" "$IFS" "$a" ""'
  ' '$'\t\n'
  éé​
  ''
  $ a=$'\u00e9e\u301\u200b' zsh -c 'LC_ALL=C printf "%q\n" "$IFS" "$a" ""'
  \ $'\t'$'\n'$'\0'
  $'\303'$'\251'e$'\314'$'\201'$'\342'$'\200'$'\213'
  ''
  $ a=$'\u00e9e\u301\u200b' bash -c 'LC_ALL=C printf "%q\n" "$IFS" "$a" ""'
  $' \t\n'
  $'\303\251e\314\201\342\200\213'
  ''
  

• q, qq, qqq, qqqq parameter expansion flags in zsh.

  for various types of quoting, qqqq being the one for $'...':

  $ a=$'\u00e9e\u301\u200b' zsh -c 'print -r -- ${(qqqq)a}'
  $'éé​'
  $ a=$'\u00e9e\u301\u200b' zsh -c '(){local LC_ALL=C; print -r -- ${(qqqq)a}}'
  $'\303\251e\314\201\342\200\213'
  

  There's also q and q+ that only uses quoting for things that need it (though still with the caveat for those unicode ones).

various non-standard commands:

• hex-dumper: hexdump, hd, xxd... You'd want to feed them the output of printf %s "$var" (or print -rn -- "$var" with ksh/zsh, or echo -nE - "$var" with zsh...).

• cat -vte or cat -A

• uconv -x hex for unicode code points of characters (as opposed to hex value of the bytes of the encoding), only for UTF-8 (one can preprocess the input with iconv -t utf-8 though provided it's valid text in the locale's encoding)

• uconv -x name for the character names

• recode ..dump. both hex and name but know about fewer Unicode characters (not updated with newer versions of Unicode). Works in non-UTF-8 locales though.

Answer 2

Especially with IFS, you absolutely want to quote it, since otherwise it, well turns to nothing. You did that already, so no problem there.

As for echo, it depends on the shell. Some versions of echo handle backslash-escapes by default, some don't. Bash's doesn't, zsh's does:

$ bash -c 'echo "foo\nbar"'
foo\nbar
$ zsh -c 'echo "foo\nbar"'
foo
bar

It's better to use printf instead: printf "%s" "$IFS" | hexdump -C.

See also: Why is printf better than echo?

printf "%q" "$IFS" also works in Bash and zsh.

That should keep you fine, except that Bash can't handle NUL bytes (\0) at all, zsh can. Bash:

$ var=$'foo\0bar'
$ printf "%q\n" "$var"
foo

zsh:

$ var=$'foo\0bar'
$ printf "%q\n" "$var"
foo$'\0'bar

Answer 3

The only character that doesn't work in bash is the null.

$ var="$(perl -wE 'print map chr, 0 .. 255')"
$ echo -n "$var" | xxd
0000000: 0102 0304 0506 0708 090a 0b0c 0d0e 0f10  ................
0000010: 1112 1314 1516 1718 191a 1b1c 1d1e 1f20  ............... 
0000020: 2122 2324 2526 2728 292a 2b2c 2d2e 2f30  !"#$%&'()*+,-./0
0000030: 3132 3334 3536 3738 393a 3b3c 3d3e 3f40  123456789:;<=>?@
0000040: 4142 4344 4546 4748 494a 4b4c 4d4e 4f50  ABCDEFGHIJKLMNOP
0000050: 5152 5354 5556 5758 595a 5b5c 5d5e 5f60  QRSTUVWXYZ[\]^_`
0000060: 6162 6364 6566 6768 696a 6b6c 6d6e 6f70  abcdefghijklmnop
0000070: 7172 7374 7576 7778 797a 7b7c 7d7e 7f80  qrstuvwxyz{|}~..
0000080: 8182 8384 8586 8788 898a 8b8c 8d8e 8f90  ................
0000090: 9192 9394 9596 9798 999a 9b9c 9d9e 9fa0  ................
00000a0: a1a2 a3a4 a5a6 a7a8 a9aa abac adae afb0  ................
00000b0: b1b2 b3b4 b5b6 b7b8 b9ba bbbc bdbe bfc0  ................
00000c0: c1c2 c3c4 c5c6 c7c8 c9ca cbcc cdce cfd0  ................
00000d0: d1d2 d3d4 d5d6 d7d8 d9da dbdc ddde dfe0  ................
00000e0: e1e2 e3e4 e5e6 e7e8 e9ea ebec edee eff0  ................
00000f0: f1f2 f3f4 f5f6 f7f8 f9fa fbfc fdfe ff    ...............

printf is more portable then echo, but for my system and shell (bash), their output is exactly the same.

printf %s "$var"

Answer 4

I love the Q parameter transformation operator in Bash:

echo "${IFS@Q}"

Output:

$' \t\n'

This $'string' quoting is called ANSI-C Quoting.

The Q operator can also be used to pass parameters to commands with SSH:

f='filename with space'
ssh "$host" "ls -l ${f@Q}"

The Parameter Expansion section has the list of all operators. The A operator is also useful:

a=(array with 'various elements')
echo "${a[@]@A}"

Output:

declare -a a=([0]="array" [1]="with" [2]="various elements")