wget with URL variables?

Question

I want know if I can use variables inside the url used by wget? I'm trying to recreate in BASH: I currently have:

WK_ARTIST="$(echo $ARTIST | sed 's/\ /\_/g' )"
WK_ALBUM="$(echo $ALBUM | sed 's/\ /\_/g' )"
echo "$WK_ARTIST"
echo "$WK_ALBUM"
wget https://en.wikipedia.org/wiki/File:"$WK_ARTIST"-"$WK_ALBUM".jpg

I get

Warning: wildcards not supported in HTTP.
--2021-06-10 07:24:20--  https://en.wikipedia.org/wiki/File:Lamb_Of_God_-_%1B[1;36mLamb_Of_God.jpg

Can this be done.

Pleae add echo "ARTIST=$ARTIST, ALBUM=$ALBUM" and its result to your question — Chris Davies, Jun 10 '21 at 10:21
Does this answer your question? Why does my shell script choke on whitespace or other special characters? — Chris Davies, Jun 10 '21 at 10:28
@roaima quoting will solve it, of course, but whitespace isn't the issue here. It will also fail if the values have no whitespace since the shell is looking for $ARTIST_ here instead of $ARTIST. — terdon, Jun 10 '21 at 10:39
@terdon In the current installment of the question, quoting is not an issue (wel, not in the wget bit anyway). However, it would be good to know what exactly is in those variables and where that data comes from. It looks as if the ARTIST variable may possible contain junk at the start. — Kusalananda, Jun 16 '21 at 09:33
By inserting a line set -x into your script or running it like bash -x yourscript you should be able to debug similar problems. — Bodo, Jul 13 '21 at 17:26

cas · Accepted Answer · 2023-07-12T15:04:33.443

WK_ARTIST="$(echo "$ARTIST" | sed 's/ /_/g' )"
WK_ALBUM="$(echo "$ALBUM" | sed 's/ /_/g' )"
echo "$WK_ARTIST"
echo "$WK_ALBUM"

wget "https://en.wikipedia.org/wiki/File:${WK_ARTIST}_-_$WK_ALBUM.jpg"

Shell variables should be quoted whenever you use them (including inside command substitution). The sole exception is when you know with absolute certainty that you actually want the variable's value to be subject to shell's whitespace splitting.
$WK_ARTIST needs to be disambiguated from the _ by surrounding it with curly braces ({}) so that the shell doesn't interpret the variable name as $WK_ARTIST_ (which probably does not exist so evaluates as the empty string).

This is necessary for WK_ARTIST but not later in the string for WK_ALBUM because _ is a valid character in a shell variable name (just like A-Z, a-z. and 0-9 for characters after the first), while . is not.

i.e. FOO, FOO_, FOO1, FOO_1, and FOOBAR are all valid and completely distinct variable names.

Or, using bash's built-in search & replace feature instead of echo ... | sed:

WK_ARTIST="${WK_ARTIST// /_}"
WK_ALBUM="${WK_ALBUM// /_}"
echo "$WK_ARTIST"
echo "$WK_ALBUM"
wget "https://en.wikipedia.org/wiki/File:${WK_ARTIST}-$WK_ALBUM.jpg"

Or, if you're going to use the URL multiple times (or just to make the script easier to read), it's worth doing something like:

...
URL="https://en.wikipedia.org/wiki/File:${WK_ARTIST}_-_$WK_ALBUM.jpg"
...
wget "$URL"

Thnx so much.I'm really looking at things much better now. – Dark_Stoner Jun 15 '21 at 18:01 — Dark_Stoner, Jun 15 '21 at 18:01

score 1 · Answer 2 · answered Jun 10 '21 at 10:37

1

You can use variables, yes, but you need to quote them so the shell can know where the variable name ends:

$ ARTIST=foo
$ ALBUM=bar
$ echo $ARTIST_-_$ALBUM.jpg
-_bar.jpg

So, the shell is trying to expand a variable called $ARTIST_, which doesn't exist. But, if you quote:

$ echo "$ARTIST"_-_"$ALBUM".jpg
foo_-_bar.jpg

So, try:

wget https://en.wikipedia.org/wiki/File:"$ARTIST"_-_"$ALBUM".jpg

answered Jun 10 '21 at 10:37

terdon

242,166

Thanks but I've already tried that and still get the same error – Dark_Stoner Jun 10 '21 at 10:49
2

@Dark_Stoner that seems unlikely, I just tried and could download the file perfectly well with ARTIST=Lamb_of_God; ALBUM=Lamb_of_God; wget https://en.wikipedia.org/wiki/File:"$ARTIST"_-_"$ALBUM".jpg. Please edit your question and show us how you define your variables as Roaima asked. – terdon Jun 10 '21 at 10:51
I see my error is due to a color code injected by grep! – Dark_Stoner Jun 10 '21 at 11:46
My error is that I didn't read the complete error. A color code got introduced into the code because I used grep to filter out the tuxi results. – Dark_Stoner Jun 10 '21 at 11:51
Ah yes, now that you show the full error that can be seen. So, just don't use color with grep. Either use /bin/grep instead of grep (many Linux systems have grep aliased to grep --color=always) or explicitly turn the color off with grep --color=no. – terdon Jun 10 '21 at 12:02

wget with URL variables?

2 Answers2