curl --data-binary $' inside a shell script

Question

I have tried everything to do a curl from within a shell script. This curl has a

--data-binary $'

part which I just can not get my script to recreate. It will move the $ inside the single quote for some weird reason. Any ideas how to fix it?

--data-binary '$

curl -s  -H 'Connection: keep-alive' \
  -H 'Cache-Control: max-age=0' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'Origin: https://url.com' \
  -H 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundary6AiBMw0TyIybreRa' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-User: ?1' \
  -H 'Sec-Fetch-Dest: document' \
  -H 'Referer: https://url.com' \
  -H 'Accept-Language: en-US,en;q=0.9,es;q=0.8' \
  -H "Cookie: JSESSIONID=$jsession; __utmc=35664418; __utmz=35664418.1593348512.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); BALANCEID=balancer.apphpgal01; acceta_default=im10snhiask7qic85vrr1dr4i2; __utma=35664418.1302973562.1593348512.1593372701.1593378049.6; __utmt=1; __utmb=35664418.5.10.1593378049" \
  --data-binary $"------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name=\"$var1\"\r\n\r"$var2"\r\n------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name=\"$var3\"\r\n\r\n\r\n------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name=\"g-recaptcha-response\"\r\n\r\n$recaptcha\r\n------WebKitFormBoundary6AiBMw0TyIybreRa--\r\n" \
  --compressed "https://url.com"

It is driving me crazy. I have tried everything. Escaping the $, putting it in single quotes. Double quotes. Changing every " for '. I'm definitely missing something guys.

EDIT: Someone pointed out that I'm not explaining myself that clearly. Rightly so: I'm running the script like this "bash curl.sh". The content of curl.sh would be

curl -s -H "Cookie: JSESSIONID=$jsession" \
  --data-binary $"------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name=\"$var1\"\r\n\r"$var2"\r\n------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name=\"g-recaptcha-response\"\r\n\r\n$recaptcha\r\n------WebKitFormBoundary6AiBMw0TyIybreRa--\r\n" \
  --compressed "https://url.com"

However when I execute it, the --data-binary part puts the dollar sign $ inside the quotes

I have also tried this per suggestion of @Stéphane Chazelas without success

--data-binary \$'------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name="'"$var1"'"\r\n\r'"$var2"'\r\n------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name="'"$var3"'"\r\n\r\n\r\n------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name="g-recaptcha-response"\r\n\r\n'"$recaptcha"'\r\n------WebKitFormBoundary6AiBMw0TyIybreRa--\r\n' \
  --compressed "https://url.com"

Will run

--data-binary '$------WebKitFormBoundary

Same thing happens with

--data-binary "$"'------WebKitFormBoundary

as well as

--data-binary '$''------WebKitFormBoundary

Well, first, you don't have $' or '$ in that command you've shown. Second, what do you mean it gets moved inside the single quote? What's the value you're actually trying to give to curl? What's the value it's getting? — ilkkachu, Jun 29 '20 at 17:22
Ah, right. Can you show exactly how you're running that script? And can you show the exact contents of the script? With the whole curl command changed to something shorter to demonstrate the problem, e.g. just an echo? — ilkkachu, Jun 29 '20 at 17:25
I never wrote \$'...' in my answer. Do you want a literal $ to be included in the argument passed to curl, or do you want to use the $'...' quoting operator to allow the \r to be expanded to a CR character? — Stéphane Chazelas, Jun 30 '20 at 14:27

Stéphane Chazelas · Answer 1 · 2020-07-03T09:25:15.210

The $"..." ksh93 quoting operator (also found in bash) is for internationalising messages, it's not what you want here.

To expand the \r\n, you need $'...' (also from ksh93 but more widely supported).

Parameters are not expanded within though, so you'd need:

--data-binary $'....\r\n...name="'"$var1"$'"...\r\n...'

For --data-binary and ....<CR><LF>...name="<contents-of-var1>"...<CR><LF>... (where <CR>, <LF>, <contents-of-var1> stand respectively for the Carriage Return control character, the Line Feed aka Newline control character and the contents of the $var1 variable) arguments to be passed to curl.

That is, use $'...' for the parts where you want \x sequences to be expanded and "..." for the parts that need variables to be expanded.

Another option is to define:

CR=$'\r' LF=$'\n' CRLF=$'\r\n'

Or using standard sh syntax:

eval "$(printf 'CR="\r" LF="\n" CRLF="\r\n"')"

and use standard "..." for the whole thing:

--data-binary "...${CRLF}...name=\"$var1\"..."

See How to use a special character as a normal one? for details on various quoting operators in various shells.

Hello @Stéphane, I have tried some of what you stated as well as the linked on your answer.
Executing --data-binary \$'------WebKitFormBoundary

returned --data-binary '$------WebKitFormBoundary — user1079378, Jun 30 '20 at 09:37

ilkkachu · Answer 2 · 2020-06-30T12:14:11.493

So, what I think you want is basically this:

$ cat quote.sh
#!/bin/bash
str=foobar
printf "%s\n" $'str='"$str"$'\nanother line\n'
#             <-----><----><----------------->  see below
$ bash quote.sh
str=foobar
another line

I.e. the \r\n changed to CRLF, and the variables expanded. As Stéphane mentioned, you have to use both $'...' and "..." for that. $'...' interprets the backslash-escapes and the regular double quotes expand variables within. Neither does both, and $"..." is different still (and not useful here). I marked the three distinct quoted strings above. Note that if you need to put literal double-quotes in the string going to curl, you can just put them as-is within single quotes. $'"\n"' is two double-quotes separated by a newline.

I asked about running the script, because $'...' isn't a standard feature, and not supported in e.g. Dash, which Debian and Ubuntu have as /bin/sh. So using that shell would fail:

$ sh quote.sh
$str=foobar$\nanother line\n

The result is a bit similar to what you'd get in Bash if you put the $ inside the single quotes, like echo '$str=...'.

In all of these, the $ is quoted or escaped, and gets interpreted as a regular character, not as the start of the $'...' quoting.

--data-binary \$'------Web...'
--data-binary '$------Web...'
--data-binary "$"'------Web...'
--data-binary '$''------Web...'

And if you used any of those with set -x in effect, Bash would indeed output the resulting string as '$----...':

$ echo --data-binary \$'------Web...'
+ echo --data-binary '$------Web...'
--data-binary $------Web...

Actually, since that string is rather unwieldy and over multiple lines, you might also want to build it in pieces.

Something like this:

data=
data+=$'------WebKitFormBoundary6AiBMw0TyIybreRa\r\n'
data+=$'Content-Disposition: form-data; name="'"$var1"$'"\r\n'
#      ^                                     *^^     ^^^*    ^
#   ^ = shell quotes, * = quotes in the resulting string
data+=$'------WebKitFormBoundary6AiBMw0TyIybreRa\r\n'
...

and then

curl ... --data-binary "$data"

curl --data-binary $' inside a shell script

2 Answers2