5

I find that when writing text as input to another program, any command substitutions in double quotes within the intended text are interpreted and expanded by the shell

The links in the answer here states that single quotes can be used to prevent parameter expansion or command substitution. However I'm finding that enclosing a command substitution in single-quotes also fails to stop the shell from expanding the command substitution

How do you prevent the shell from interpreting command substitutions that are intended as text rather than a command to be executed?

A demonstration

$ echo "`wc -l *`"

attempts to count lines in all files in the current directory

$ echo "'`wc -l *`'"

Same result, i.e. counts lines in all files in the current directory

update From this demonstration I've spotted that the problem seems to be that I am quoting the single quotes. I think enclosing single quotes and ` (backtick) in double quotes preserves the literal meaning of (i.e. suppresses) the single quotes but does not preserve the literal meaning of the backquote (i.e. backtick) that introduces the command substitution.

In my use case the input for another command needs to be quoted. With this document saying that:

A single-quote cannot occur within single quotes

How do you prevent a single-quoted command substitution from being expanded when the single-quoted command substitution is within a (double) quoted string? There should be a way to do it other than using backslash escapes

Actual situation

In a program I'm using the only way to split a description of a task into separate lines is to enclose the description in double-quotes:

$ task add "first line doesn\'t say much
Second line says a lot but part of this line does not appear in the resulting description 'truncate -s0 !(temp_file | temp_dir)' truncates all files to 0 bytes as shown by: '`wc -l *`'"

The resulting description:

first line doesn\ -s0 !(temp_file | temp_dir)' truncates all files to 0 bytes as shown by: 0 file1 10 file2 0 directory1 0 directory2 502 file3 123 file4 162 file5 0 directory3

As you can see 

't say much
Second line says a lot but part of this line does not appear in the resulting description 'truncate

is missing from the description and the shell has interpreted 'wc -l *' as a command substitution, thereby including the line counts of all files in the current directory as part of the description

What's causing the shell to remove the part of the argument to task between \ (backslash) and -s, and how do you prevent the shell from interpreting the above single-quoted command substitution (i.e. '`wc -l *`')?

bit
  • 1,106
  • 4
    You mean echo '$(ls)' does not produce $(ls) as its output? Can you add the example that is not working for you in the question? – NickD Jul 29 '19 at 20:20
  • Note that adding single quotes within double quotes won't help. – Andy Dalton Jul 29 '19 at 20:42
  • @NickD good suggestion, I've updated the question – bit Jul 29 '19 at 20:55
  • Can you not just use single quotes instead of double quotes? – Andy Dalton Jul 29 '19 at 20:55
  • @AndyDalton I updated the question with this discovery. What's the alternative other than using backslash escapes? – bit Jul 29 '19 at 20:56
  • Do you mean using two single quotes like $ echo ''`wc -l *`''. It gives the same result, i.e. tries counting lines in all files in the current directory – bit Jul 29 '19 at 21:00
  • No: echo '\wc`'` – Andy Dalton Jul 29 '19 at 21:05
  • The echo demonstration is an example. In my use case the quoted command substitution is in a quoted argument to an external command. The argument to the external command needs to be quoted in other for that particular command to work. Imagine that echo '`wc -l *`' is actually within a quoted argument to another command, i.e. something like $ anotherCommand "echo '`wc -l *`'" – bit Jul 29 '19 at 21:12
  • 4
    You know, instead of us guessing, you should put your actual example in the question. – NickD Jul 30 '19 at 00:19
  • "There should be a way to do it other than using backslash escapes" Yet there isn't. You should use backslashes, or simply end/start your double quotes. bar=1; quux=2; echo "foo $bar "'\wc -l *`'" $quux etc"`. –  Jul 30 '19 at 08:47
  • 2
    @MyWrathAcademia, like NickD said, please [edit] your question to show the actual situation, not something where you have removed parts of the scenario. Are you running something like eval in anotherCommand? Or passing the command over SSH? Or just expanding a variable containing the command? Things like that matter. – ilkkachu Jul 30 '19 at 10:19
  • @ilkkachu , I'm not sure the real scenario would make a difference because the example I used is very close to the actual problem I'm dealing with. If someone can solve the echo example without using backslashes then the solution would easily apply to the actual situation. To answer your question, I'm not executing eval in anotherCommand and although I've not looked at the internals of anotherCommand I don't think it's executing eval either – bit Jul 30 '19 at 18:45
  • @ilkkachu I've put the actual scenario in the question as first requested by NickD – bit Jul 30 '19 at 21:37
  • What is the task add ? An script, a command ? Could you provide or link to its internal description ? –  Jul 31 '19 at 17:57
  • Without having the internals of task the only thing I can guess is that there is some double shell expansion which is affected by two different levels of quoting. –  Jul 31 '19 at 17:59
  • 1
    Note that a shell string is either single quoted or double quoted, whichever statrted the quoting space. There is no such thing as joint quoting (both double and single quotes in effect at the same time). –  Jul 31 '19 at 18:02

4 Answers4

11

Use single-quote strong quoting:

printf '%s\n' '`wc -l *`'

And if you want to also include single quotes in that argument passed to printf, you'd need to use different quotes for ' itself like:

printf '%s\n' '`wc -l *` and a '"'"' character'

Or:

printf '%s\n' '`wc -l *` and a '\'' character'

Other alternatives include escaping the ` with backslash inside double quotes:

printf '%s\n' "\`wc -l *\` and a ' character"

Or have ` be the result of some expansion:

backtick='`'
printf '%s\n' "${backtick}wc -l *${backtick} and a ' character"

Also note:

cat << 'EOF'
`wc -l *` and a ' character and a " character
EOF

to output arbitrary text without having to worry about quoting (note the quotes around the first EOF).

You can also do:

var=$(cat << 'EOF'
echo '`wc -l *`'
EOF
)

Which with ksh93 or mksh you can optimise to:

var=$(<<'EOF'
echo '`wc -l *`'
EOF
)

(also works in zsh, but still runs cat in a subshell there) for $var to contain literally echo '`wc -l *`'.

In the fish shell, you can embed ' within '...' with \':

printf '%s\n' '`wc -l *` and a \' character'

but anyway ` is not special there, so:

printf '%s\n' "`wc -l *` and a ' character"

would work as well.

In rc, es or zsh -o rcquotes, you can insert a ' within '...' with '':

printf '%s\n' '`wc -l *` and a '' character'

See How to use a special character as a normal one? for more details.

Stéphane Chazelas
  • 544,893
  • In your first example that includes a single-quote in printf's argument why are you single-quoting the single quote that you want it's literal meaning to be preserved when you already double quoted it? Similarly in your second example that includes a single-quote in the argument for printf why are you single-quoting a single-quote that has already been escaped using a backslash? Isn't the single-quote in your examples that I highlighted above redundant? – bit Jul 30 '19 at 19:38
  • Your trick to prevent the shell from interpreting special characters in a here document is very useful. And your solution to store the result of the command substitution in a variable is also immensely useful. Can you explain why the value of $var is '`wc -l *`' where as $ echo $var outputs echo '`wc -l *`'? Overall this answer is a wealth of wisdom – bit Jul 30 '19 at 20:05
  • I'm not really familiar with variable expansion, why not printf '%s\n' "$backtick wc -l * $backtick and a ' character"? Does ${backtick} prepend text when in front of a string and append text when behind a string? In other words, can you clarify how ${backtick} works? I'm pleasantly surprised and intrigued that variable expansion (if that's the right word) prevents the shell from interpreting special characters. Does this apply to all special characters not just the backtick? May be using variable expansion to prevent interpretation of special characters by the shell is exactly what I need – bit Jul 30 '19 at 20:11
  • Please edit your answer to include the actual situation that I added to my question because the variable expansion you suggested is not working – bit Jul 30 '19 at 21:39
  • @MyWrathAcademia see LESS='+/^SIMPLE COMMAND EXPANSION' man bash and also LESS='+/^EXPANSION' man bash for complete details. In particular notice which types of expansion (in the latter doc section) are noted together in precedence; command substitution and parameter expansion have equal precedence, so neither form of expansion can cause the other (i.e. myvar='$(ls)'; echo $myvar doesn't call ls). – Wildcard Jul 30 '19 at 21:45
  • 1
    @Wildcard Operator precedence imply that both operators are used (executed), like 2+34 becomes 14, both sum and multiply are executed. In this case, the expansion of $ only happens once. That is: it shaves one* level of $ expansion. That is: a var expansion doesn't happen after a com expansion (try echo $(echo '$a')) nor a com expansion will happen after a var expansion (you posted it already). That is not true inside an arithmetic expansion, try a=23 b=a c=b d=c e=$((d)); echo "a=$a b=$b c=$c d=$d e=$e" it keeps expanding vars as long as there is a valid var name to expand. –  Jul 31 '19 at 17:50
4

Here (linebreaks added),

$ task add "first line doesn\'t say much
Second line says a lot but part of this line does not appear in the
resulting description 'truncate -s0 !(temp_file | temp_dir)' truncates
all files to 0 bytes as shown by: '`wc -l *`'"

the whole string is double-quoted, so command substitutions and other expansions will run there. That happens in the shell, before task sees that string, and you'll need to prevent it with backslashes or putting that part in single quotes.

E.g. 

$ printf "%s\n" "...shown by: '\`wc -l *\`'"
...shown by: '`wc -l *`'

So, 

task add "...shown by: '\`wc -l *\`'"

would pass the string ...shown by: '`wc -l *`' to task. It's up to it what does with that.

If you don't want to use backslashes, here's the way to put it in single quotes:

#               aaaaaaaaaaaaaaaaBBBBBBBBBBBaaa
$ printf "%s\n" "...shown by: '"'`wc -l *`'"'"
...shown by: '`wc -l *`'

(The a's mark the double-quoted parts, the B's the single-quoted parts. They are just concatenated on the shell command line. The literal single quotes are within the double-quoted strings.)

As for the single quote and the backslash, you don't need to escape a single quote within double quotes, and in fact the backslash will remain there:

$ printf "%s\n" "foo'bar"
foo'bar
$ printf "%s\n" "foo\'bar"
foo\'bar

From what you show, it seems like task removes at least the first single-quoted string from the argument (plus a word after that, since the removed part was 't say much ... 'truncate)

The shell will not do that, this works fine:

$ printf "%s\n" "a 'quoted string' to test"
a 'quoted string' to test

What's causing the shell to remove the part of the argument to task between \ (backslash) and -s,

It's highly likely it's not the shell doing that.

and how do you prevent the shell from interpreting the above single-quoted command substitution (i.e. '`wc -l *`')?

It's not single-quoted, it's double-quoted with quoted single quotes next to it.

ilkkachu
  • 138,973
  • 1
    I just got the chance to read and test your answer. It works finely; using string concatenation to put the part whose literal meaning is not preserved by double-quotes inside single-quotes is very useful. Although I noticed unexpected behavior in the string concatenation: I assume an implicit + (plus) operator is between the strings being combined so why is it that when there is no space between strings the new string created from the combined strings has no space where they were combined but if you separate the strings with a space then the new string has a space where they were combined? – bit Aug 05 '19 at 19:08
  • @MyWrathAcademia, hmm, separate with a space how? If you write something like somecmd "foo" 'bar', then there's actually two arguments (in the same way as writing ls -l foo.txt passes the option and the filename as separate arguments to ls.) Though if you happen to test with echo, you won't see it, since it combines the arguments with a single space... – ilkkachu Aug 05 '19 at 19:10
  • 1
    What I mean is, task add "...shown by:'"'`wc -l *`'"'" creates the string "...shown by:'`wc -l *`' (i.e. no white space character in between characters where the strings were combined) where as task add "...shown by:'" '`wc -l *`' "'" creates the string "...shown by:' `wc -l *` ' (i.e. white space character in between the characters where the strings were combined) – bit Aug 05 '19 at 19:35
  • @MyWrathAcademia, yes, that's exactly what I mean, the latter doesn't create one string, it creates three. If the command itself then concatenates them with a space, well, that's another thing. But try with something that does otherwise: printf ":%s:\n" "...shown by:'" '\wc -l *`' "'"` vs. the one without spaces. (retake, I messed up the backticks) – ilkkachu Aug 05 '19 at 19:47
1

You can escape the backticks by using a backslash as shown below:

echo "\`wc -l *\`"
justsomeguy
  • 368
  • I mentioned prefering solving the problem without using backticks because you would need to escape every character you don't want to be interpreted by the shell where as if quoting could solve the problem you could use one single-quote to quote a substring that contains special characters such as $, ` and \ that you don't want to be interpreted by the shell – bit Jul 29 '19 at 21:23
  • ok, so the reason you need to use double quotes is because you have a variable that needs to be interpreted as well? So, as a result, echo 'wc -l *' wont work as solution? – justsomeguy Jul 29 '19 at 22:02
0

To escape any arbitrary character sequence so that no special characters are processed and no substitution is done:

  1. Replace every ' with '\''
  2. Enclose the resulting string in single quotes.

For example:

`wc -l 'my file.txt'`

Becomes

'`wc -l '\''my file.txt'\''`'

Note if the string starts or ends with ', you end up with '' at the start/end, which does nothing and can be removed but it's valid to leave it there (e.g. to keep a string-escaping function simple).

Keiji
  • 391