Shell-independent escaping for strings in scripts

Question

I just struggled (again) over this:

# only in bash
NORM=$'\033[0m'
NORM=$'\e[0m'

# only in dash
NORM='\033[0m'

# only in bash and busybox
NORM=$(echo -en '\033[0m')

The goal is to include special characters in the string, not only for output using echo but also for piping into a cli tool etc.

In the specific use case above using $(tput ...) is probably the best way, but I ask for a general escaping solution with minimal requirements to external tools but maximum compatibility.

Normally, I help myself using conditions like [ -z "$BASH_VERSION" ] but

1. I didn't find an easy way to detect busybox yet
2. a normal variable assignment in 5 lines (using if/else/fi) looks like overkill
3. I prefer simple solutions

Answer 1

What you want is "$(printf ...)". Stephane has already written an excellent expose of printf vs echo, more of an article than a mere answer, so I won't repeat the whole thing here. Keynotes pertinent to the current question are:

1. Stick to POSIX features and it is very portable, and
2. It is frequently a shell builtin, in which case you have no external calls or dependencies.

I will also add that it took me quite a while (okay, just a few weeks) to get around to switching from echo, because I was used to echo and thought printf would be complicated. (What are all those % signs about, huh?) As it turns out, printf is actually extremely simple and I don't bother with echo anymore for anything but fixed text with a newline at the end.

---

Printf Made Easy

There are vast numbers of options for printf. You can print numbers to specific decimal places of accuracy. You can print multiple fields, each with a specified fixed width (or a minimum width, or a maximum width). You can cause a shell string variable which contains the character sequences \t or \n to be printed with those character sequences interpreted as tabs and newlines.

You can do all these things, and you should know they are possible so you can look them up when you need them, but in the majority of cases the following will be all you need to know:

printf takes as its first argument a string called "format". The format string can specify how further arguments are to be handled (i.e. how they will be formatted). Further arguments, if not referenced at all* within the format argument, are ignored.

Since alphanumeric characters (and others) can be embedded in the format argument and will print as-is, it may look like printf is doing the same thing as echo -n but that for some unknown reason it's ignoring all but the first argument. That's really not the case.

For example, try printf some test text. In this example some is actually taken as the format, and since it doesn't contain anything special and doesn't tell printf what to do with the rest of the arguments, they are ignored and all you get printed is some.

% followed by a specific letter needs to be used within the format string (the first argument to printf) to specify what type of data the subsequent arguments contain. %s means "string" and is what you will use most often.

\n or \t within the format translate to newlines and tab characters respectively.

That's really all you need to use printf productively. See the following code block for some very simple illustrative examples.

$ var1="First"
$ var2="Second"
$ var3="Third"
$ printf "$var1" "$var2" "$var3"       # WRONG
First$                                 # Only the first arg is printed, without a trailing newline
$
$ printf '%s\n' "$var1"                # %s means that the next arg will be formatted as a literal string with any special characters printed exactly as-is.
First
$
$ printf '%s' "$var1" "$var2" "$var3"  # When more args are included than the format calls for, the format string is reused.  This example is equivalent to using '%s%s%s' as the format.
FirstSecondThird$                      # All three args were printed; no trailing newline.
$
$ printf '%s\t%s\t%s\n' "$var1" "$var2" "$var3"
First   Second  Third                  # Tab separation with trailing newline.  This format is very explicit about what to do with three arguments.  Now see what happens if four are used:
$ var4="Fourth"
$ printf '%s\t%s\t%s\n' "$var1" "$var2" "$var3" "$var4"
First   Second  Third             # The specified format is reused after the three expected args,
Fourth                            # so this line has two trailing tabs.
$
$ printf '%s\n' "$var1" "$var2" "$var3"  # This format reuse can be used to advantage in printing a list, for example.
First
Second
Third
$
$ printf '%s\t' "$var1" "$var2" "$var3" ; printf '\n'  # Here is a dual command that could have args added without changing the format string...
First   Second  Third   
$ printf '%s\t' "$var1" "$var2" "$var3" "$var4" ; printf '\n'
First   Second  Third   Fourth              # ...as you can see here.
$                                           # It does print a trailing tab before the newline, however.

_{* Of course, if you include a single argument format specifier sequence such as %s, your whole format string is reused as many times as needed to handle all arguments provided. See examples.}

Answer 2

printf is good for printing arbitrary octals and so on, but you're just trying to enter ESC - so you can just type it. If you're worried about your terminal eating the escape you can do CTRL+V first.

and so...

NORM='^[[0m'

...is what you'll likely see displayed on your screen after typing in the sequence described above because the tty will quote the output of non-printable characters that way, but the shell will read the literal ESC character in the ^[ escape sequence's place.

This capability is independent of any terminal emulator program or shell or what-have-you - though it is configurable. The system kernel manages your input as configured by the stty utility. The default settings - on a BSD, Linux, or (as far as I'm aware) nearly any other unix-like system - typically include most of the following:

stty --help 2>&1 | sed -e '/Special/,$!d;/./!q'

---

Special characters:
 * discard CHAR  CHAR will toggle discarding of output
   eof CHAR      CHAR will send an end of file (terminate the input)
   eol CHAR      CHAR will end the line
 * eol2 CHAR     alternate CHAR for ending the line
   erase CHAR    CHAR will erase the last character typed
   intr CHAR     CHAR will send an interrupt signal
   kill CHAR     CHAR will erase the current line
 * lnext CHAR    CHAR will enter the next character quoted
   quit CHAR     CHAR will send a quit signal
 * rprnt CHAR    CHAR will redraw the current line
   start CHAR    CHAR will restart the output after stopping it
   stop CHAR     CHAR will stop the output
   susp CHAR     CHAR will send a terminal stop signal
 * swtch CHAR    CHAR will switch to a different shell layer
 * werase CHAR   CHAR will erase the last word typed

---

The lnext character is the one typically configured for CTRL+V. It can come in pretty handy - you can do CTRL+V then CTRL+J, for example, to insert a newline into the input buffer before the shell gets a chance to read it. Try it - and you'll probably notice that you don't see the shell's $PS2 prompt as you otherwise would. For shells that hijack the TAB key you can usually still enter one literally into the command-line by prefixing the keypress with CTRL+V. You can enter literal CTRL+D EOF characters, and literal CTRL+C INT characters (if not using zsh) in the same way.

Most of these characters are not even special to shell in any way, by the way, and when literally entered don't even really need quoting (though a newline would, of course).

Do stty -a to get a clear look at how your terminal is configured - chances are pretty good it's configured much the same as mine and all the rest.

And if you literally enter an ESC character into a shell-script while editing it, it remains in the file as is - the literal character is there where you put it each time the script is called, and doesn't need expansion or other interpretation to make it mean what it should.