Curl a string with whitespaces and characters in it?

Question

I'm trying to create a "simple" script that I'm going to use to create some documentation for Unix/Mac machines. I've successfully received the information I need, but it seems like I can't send it over to a webserver I've created with CURL because it gives some whitespaces and characters in the information output. I've tried with very many methods but nothing seems to work correctly..

This is the URL output that the script will send over to curl:

"ip.ip.ip.ip/documentation/index.php?avdelning=XX&user=XXX&hostname=XXX&model=ModelName:MacminiModelIdentifier:Macmini4,1&RAM=Memory:8GB&CPU=ProcessorName:IntelCore2DuoProcessorSpeed:2.4GHzNumberofProcessors:1&CPUCORES=TotalNumberofCores:2&MACADDRESS=etherxx:xx:xx:xx:xx:xx&SERIALNUMBER=SerialNumber(system):XXXXXXXX&LASTLOGGEDIN=XXXXX

Using only the following string will work if I do it directly with CURL and will not give me any errors and I receive the information on the webserver.

But when I use the script it does not work. The script:

#!/bin/bash

STR_RAM=$(/usr/sbin/system_profiler SPHardwareDataType | grep Memory: | sed 's/ //g');
STR_MODEL=$(/usr/sbin/system_profiler SPHardwareDataType | grep Model | sed 's/ //g');
STR_CPUNAME=$(/usr/sbin/system_profiler SPHardwareDataType | grep Processor | sed 's/ //g');
STR_CPUCORES=$(/usr/sbin/system_profiler SPHardwareDataType | grep Cores: | sed 's/ //g');
STR_SERIAL=$(/usr/sbin/system_profiler SPHardwareDataType | grep Serial | sed 's/ //g');
STR_MACADDR=$(ifconfig | grep en0 -A3 | grep ether | sed 's/ //g');
STR_LOGGEDIN=$(finger | tail -1 | sed 's/ //g');
STR_HOSTNAME=$(hostname | sed 's/ //g');

STR_URL=ip.ip.ip.ip/documentation/index.php?avdelning=$1\&user=$2\&hostname=$STR_HOSTNAME\&model=$STR_MODEL\&RAM=$STR_RAM\&CPU=$STR_CPUNAME\&CPUCORES=$STR_CPUCORES\&MACADDRESS=$STR_MACADDR\&SERIALNUMBER=$STR_SERIAL\&LASTLOGGEDIN=$STR_LOGGEDIN;


STR_URL2=$(echo $STR_URL | sed 's/ //g' | awk '{printf("\"%s\"\n", $0);}');
curl $STR_URL2;

As you can see I've added a bunch of probably unnecessary editing with sed and awk but nothing seems to work.

I get these error messages: After I added the awk to add double quotes before and after the URL string I get:

Could not resolve "ip.ip.ip.ip

Which is very strange since I can run the curl command directly in terminal without that error. And If I do not remove the spaces etc I receive multiple lines that the URL can not be resolved. I've tried with urlencoding in perl and bash but that does not seem to fix the problem either.

I'm running this on a Mac now. Usage: sudo sh script.sh randomstring randomstring

EDIT: Output of new script

sudo sh script2.sh IT Denniz
curl: (3) Illegal characters found in URL
xx.xx.xx.xx/documentation/index.php?avdelning=XX&user=Denniz&hostname=admins-Mac-mini-2.local&model=ModelName:Macmini
ModelIdentifier:Macmini4,1&RAM=Memory:8GB&CPU=ProcessorName:IntelCore2Duo
ProcessorSpeed:2.4GHz
NumberofProcessors:1&CPUCORES=TotalNumberofCores:2&MACADDRESS=  etherXX:XX:XX:XX:XX&SERIALNUMBER=SerialNumber(system):XXXXXXXXXXX&LASTLOGGEDIN=adminadmins008Jun10:21

Exactly what I thought, that's why I added awk '{printf(""%s"\n", $0);} which did not work unforunatley, then I receive the error message that it can not resolve "ip.ip.ip.ip — Denniz, Jun 14 '16 at 14:49
Stronger than that: always quote your variables unless you know exactly what side effects you want by leaving them unquoted (and those side effects can be very subtle) — glenn jackman, Jun 14 '16 at 14:50

score 0 · Accepted Answer · edited Apr 13 '17 at 12:36

0

You don't want to insert literal quotes into the string. You just want to put quotes around the variable. I would do this:

#!/bin/bash

hw_data=$(/usr/sbin/system_profiler SPHardwareDataType)

ram=$(grep Memory: <<<"$hw_data")
model=$(grep Model <<<"$hw_data")
cpuname=$(grep Processor <<<"$hw_data")
cpucores=$(grep Cores: <<<"$hw_data")
serial=$(grep Serial <<<"$hw_data")

macaddr=$(ifconfig | grep en0 -A3 | grep ether)
loggedin=$(finger | tail -1)
hostname=$(hostname)

avdelning=$1
user=$2

url="ip.ip.ip.ip/documentation/index.php"
url+="?avdelning=${avdelning}"
url+="&user=${user}"
url+="&hostname=${hostname}"
url+="&model=${model}"
url+="&RAM=${ram}"
url+="&CPU=${cpuname}"
url+="&CPUCORES=${cpucores}"
url+="&MACADDRESS=${macaddr}"
url+="&SERIALNUMBER=${serial}"
url+="&LASTLOGGEDIN=${loggedin}"

curl "${url// /}"
#    ^          ^   <- these are the critical quotes

Read this for more about how critical quoting is: Security implications of forgetting to quote a variable in bash/POSIX shells

edited Apr 13 '17 at 12:36

Community

1

answered Jun 14 '16 at 14:55

glenn jackman

85,964

Is there a link you can provide for the different types of string expansion? i.e. single quotes vs double vs none. – eyoung100 Jun 14 '16 at 14:58
You could read from the bash manual – glenn jackman Jun 14 '16 at 14:59
Perfect... Now the proof is there for other readers. – eyoung100 Jun 14 '16 at 15:00
Thank you it seems more promising now but still it does not work! Now it gives me this error: Curl (3): Illegal characters found in URL. Tried with --globoff without succes.. Could it be (system) that's illegal? Or commas? I've tried to google which characters are disallowed without succes.. – Denniz Jun 14 '16 at 15:42
In your question, so you can properly format, show the output of echo "${url// /}" – glenn jackman Jun 14 '16 at 15:58
Alright, edited the post with the new output! Thank you so much for helping me! – Denniz Jun 14 '16 at 16:08
You need to filter the hw_data better. grep Processor is returning more than one line so you have newlines in the url – glenn jackman Jun 14 '16 at 16:20

Curl a string with whitespaces and characters in it?

1 Answers1