How can I pass argument stored on a variable to WGET

Question

I'm writing a bash script that extensively uses wget. To define all common parameters in one place I store them on variables. Here's a piece of code:

useragent='--user-agent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0"'
cookies_file="/tmp/wget-cookies.txt"
save_cookies_cmd="--save-cookies $cookies_file --keep-session-cookies"
load_cookies_cmd="--load-cookies $cookies_file --keep-session-cookies"

function mywget {
    log "#!!!!!!!!!# WGET #!!!!!!!!!# wget $quiet $useragent $load_cookies_cmd $@"  
    wget $useragent $load_cookies_cmd "$@"
}

Saddly isn't working. Somehow I'm missing the right way to store parameters on variables $useragent, $save_cookies_cmd, $load_cookies_cmd and caling wget passing these vars as parameters.

I want the result commandline as this:

wget --user-agent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0" --load-cookies /tmp/wget-cookies.txt --keep-session-cookies http://mysite.local/myfile.php

EDIT: My final solution:

In the end my script is correctly working with this:

useragent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0"
useragent_cmd="--user-agent=$useragent"
cookies_file="/tmp/wget-cookies.txt"
save_cookies_cmd="--save-cookies $cookies_file --keep-session-cookies"
load_cookies_cmd="--load-cookies $cookies_file --keep-session-cookies"

function mywget {
    log "#!!!!!!!!!# WGET #!!!!!!!!!# wget $load_cookies_cmd $useragent_cmd $@"  
    wget $load_cookies_cmd "$useragent_cmd" "$@"
}

Thanks for all your responses.

score 4 · Accepted Answer · answered Mar 19 '14 at 20:56

http://mywiki.wooledge.org/BashFAQ/050

user_agent='--user-agent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0"'
cookies_file="/tmp/wget-cookies.txt"
save_cookies_opts=( --save-cookies "$cookies_file" --keep-session-cookies )
load_cookies_opts=( --load-cookies "$cookies_file" --keep-session-cookies )

function mywget {
    log "#!!!!!!!!!# WGET #!!!!!!!!!# wget $quiet $useragent $load_cookies_cmd $@"
    wget "$user_agent" "${load_cookies_opts[@]}" "$@"
}

Note that in the wget call , "$user_agent" is quoted. This is crucial.

Clever using the array. Thanks for your response. It certainly will help me in understanding better bash script. — Filipe, Mar 20 '14 at 18:03

score 3 · Answer 2 · answered Mar 19 '14 at 20:51

3

The problem lies in the order of expansions (see EXPANSION in man bash for details). Double quotes inside a variable do not work as double quotes when the variable is expanded:

$ a='1 "2 3" 4' ; for b in $a ; do echo $b ; done
1
"2
3"
4

Note that using "$b" does not help you.

Use arrays, so you can let bash do the hard work and you can forget about quoting and escaping:

$ a=(1 '2 3' 4) ; for b in "${a[@]}" ; do echo $b ; done
1
2 3
4

answered Mar 19 '14 at 20:51

choroba

47,233

Thanks for your time. I'm kind of novice in bash script and these tricks will always help me from now on. – Filipe Mar 20 '14 at 18:05

score 3 · Answer 3 · edited Apr 13 '17 at 12:36

    mywget() ( echo log "wget $quiet \
                         userstring=${userstring:-unset} \
                         cookies=${cookies:-no} $@"
            echo wget ${userstring:+--useragent="$userstring"} \
                    ${cookies:+--${cookies}-cookies \
                    "$cookies_file" --keep-session-cookies} "$@"
    ) <<-PARAMS
            ${DQ=$(printf \\042)}
            ${NL=
}
            ${userstring="${DQ}Mozilla/5.0 \
                (Windows NT 6.1; WOW64; rv:27.0) \
                Gecko/20100101 Firefox/27.0${DQ}"}
            ${cookies_file="/tmp/wget-cookies.txt"}
            ${cookies=save}
    PARAMS

Please forgive me, but I've taken the liberty of parameterizing your function. Because the values in PARAMS will only set an unset variable, you can override their values very easily - it allows you to set sane defaults and yet still change values per function call as you like. For instance, in the above function, I've set all values to echo and here's the output when run as is:

log wget  userstring="Mozilla/5.0 \
    (Windows NT 6.1; WOW64; rv:27.0) \
    Gecko/20100101 Firefox/27.0" \
    cookies=save
wget --useragent="Mozilla/5.0 \
    (Windows NT 6.1; WOW64; rv:27.0) \
    Gecko/20100101 Firefox/27.0" \
    --save-cookies /tmp/wget-cookies.txt \
    --keep-session-cookies

But if I set "$userstring" to NULL:

    userstring= mywget
log wget  userstring=unset cookies=save
wget --save-cookies /tmp/wget-cookies.txt --keep-session-cookies

Or:

    cookies=load cookies_file=/some/other/cookie.file mywget
log wget  userstring="Mozilla/5.0 \
    (Windows NT 6.1; WOW64; rv:27.0) \
    Gecko/20100101 Firefox/27.0" \
    cookies=load
wget --useragent="Mozilla/5.0 \
    (Windows NT 6.1; WOW64; rv:27.0) \
    Gecko/20100101 Firefox/27.0" \
    --load-cookies /some/other/cookie.file \
    --keep-session-cookies

Or:

    cookies= userstring= mywget
log wget  userstring=unset cookies=no
wget

I've written more on parameter expansion here and here.

How can I pass argument stored on a variable to WGET

3 Answers3

Linked