Problem with wget and Whitespace

Question

I have a bash script that scrapes a list of urls for links to various kinds of documents. At the end, the script uses wget to download the files. However, I'm having trouble with filenames containing white space in the name: wget ends the url at the space. Is there some way to use sed or something to change the white space to %20 here? Or some other solution? This is my code:

for url in $(cat download.md)
do
    lynx --listonly --dump $url | \
awk '/\.(pdf|doc|docx|odt)$/{ sub("^[ ]*[0-9]+.[ ]*","",$0); print}'

done > ~/links.txt

for i in $( cat ~/links.txt ); do wget $i; 
done

@DopeGhoti also fails to download because the url is cut off at the space. — Linter, Aug 29 '18 at 18:58
Based on the for i in $( cat ... ) habit, I'd also recommend reading: https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice/169765#169765 — Jeff Schaller, Aug 29 '18 at 19:06
Thanks. I love how the example they laugh at is my actual code! — Linter, Aug 29 '18 at 19:09
@Linter I'd be lying if I said I never wrote similar code (before learning safer ways!) — Jeff Schaller, Aug 29 '18 at 19:50

score 1 · Accepted Answer · answered Aug 29 '18 at 18:56

1

You lost the full filenames during the $( ... ) command substitution. Just ask wget to read the file directly:

wget -i ~/links.txt

answered Aug 29 '18 at 18:56

Jeff Schaller

67,283
35
116
255

Problem with wget and Whitespace

1 Answers1