downloading a file behind a link

Question

I want to write a script to download the WEbStorm IDE from Jetbrains and then do further stuffs. But I am unable to download the tar ball.

I am following this, but it doesnt solve my problem. downloading files using wget

The webpage where I want to download the file from is https://www.jetbrains.com/webstorm/download/download-thanks.html?platform=linux

It looks like it is quite simple, but I just get a plain html page all the time when I do a wget. Doing it over the browser address bar or addons like DownThemAll is completely fine.

Following is my command

wget 'https://www.jetbrains.com/webstorm/download/download-thanks.html?platform=linux' -O WebStorm.tar.gz

I can download the file from here:

wget https://download.jetbrains.com/webstorm/WebStorm-2017.2.4.tar.gz

but I could only find this URL after a little bit of digging. Is there a way to simply get the download file url by parsing the Download button?

Is it also possible to do via curl?

"A bit of digging" means "reading the text on the download page"? — Kusalananda, Sep 12 '17 at 12:19
@Kusalananda No, a bit of digging means, using the firefox addon DownThemAll. It displays where the file is "actually" been downloaded from. — infoclogged, Sep 12 '17 at 12:22

score 1 · Answer 1 · answered Sep 12 '17 at 13:00

This link should work for latest Linux version of WebStorm: https://data.services.jetbrains.com/products/download?code=WS&platform=linux

You can get this address by calling page you wanted to get in a first place and searching for direct link:

      $ curl 'https://www.jetbrains.com/webstorm/download/download-thanks.html?platform=linux' | grep "direct link"
      <p class="sub-title no-margin-bottom">Your download should start shortly. If it doesn't, please use the <a id="download-link" href="//data.services.jetbrains.com/products/download?code=WS" data-release-download-link="">direct link</a>.</p>

Just add &platfor=your_platform and you get latest version

score -1 · Answer 2 · answered Nov 21 '17 at 04:01

I wrote a short script that will install any Linux-compatible JetBrains IDE, including WebStorm. You can find more details here.

#!/bin/sh

echo "Installing IntelliJ IDEA..."

# We need root to install
[ $(id -u) != "0" ] && exec sudo "$0" "$@"

# Attempt to install a JDK
# apt-get install openjdk-8-jdk
# add-apt-repository ppa:webupd8team/java && apt-get update && apt-get install oracle-java8-installer

# Prompt for edition
while true; do
    read -p "Enter 'U' for Ultimate or 'C' for Community: " ed 
    case $ed in
        [Uu]* ) ed=U; break;;
        [Cc]* ) ed=C; break;;
    esac
done

# Fetch the most recent version
VERSION=$(wget "https://www.jetbrains.com/intellij-repository/releases" -qO- | grep -P -o -m 1 "(?<=https://www.jetbrains.com/intellij-repository/releases/com/jetbrains/intellij/idea/BUILD/)[^/]+(?=/)")

# Prepend base URL for download
URL="https://download.jetbrains.com/idea/ideaI$ed-$VERSION.tar.gz"

echo $URL

# Truncate filename
FILE=$(basename ${URL})

# Set download directory
DEST=~/Downloads/$FILE

echo "Downloading idea-I$ed-$VERSION to $DEST..."

# Download binary
wget -cO ${DEST} ${URL} --read-timeout=5 --tries=0

echo "Download complete!"

# Set directory name
DIR="/opt/idea-I$ed-$VERSION"

echo "Installing to $DIR"

# Untar file
if mkdir ${DIR}; then
    tar -xzf ${DEST} -C ${DIR} --strip-components=1
fi

# Grab executable folder
BIN="$DIR/bin"

# Add permissions to install directory
chmod -R +rwx ${DIR}

# Set desktop shortcut path
DESK=/usr/share/applications/IDEA.desktop

# Add desktop shortcut
echo "[Desktop Entry]\nEncoding=UTF-8\nName=IntelliJ IDEA\nComment=IntelliJ IDEA\nExec=${BIN}/idea.sh\nIcon=${BIN}/idea.png\nTerminal=false\nStartupNotify=true\nType=Application" -e > ${DESK}

# Create symlink entry
ln -sf ${BIN}/idea.sh /usr/local/bin/idea

echo "Done."

The question is about how to download a file, specifically a .tar.gz file, given only the URL of the HTML web page from which it can be downloaded. Your answer assumes that the user already knows the URL of the .tar.gz file; the OP already knows how to download a file given its specific URL. And then your script does a lot of things that the question doesn’t ask for, which is OK, as long as you answer the question. Also, you should use more quotes. — G-Man Says 'Reinstate Monica', Nov 21 '17 at 04:13
The question is not specific, but I tried to solve the problem @infoclogged encountered, which involves downloading the WebStorm IDE using wget. I suspect OP is not interested in the details of wget. However even if there was a broader question here, I would note that the other answer also has the same issue (doesn't even mention wget). Also, you should use more quotes. Where should I use more quotes? Thanks! — breandan, Dec 18 '17 at 23:34
Where should I use more quotes? "$(id -u)", case "$ed" in, optionally VERSION="$(…)" (it’s OK that there are quoted strings between the parentheses), FILE=$(basename "$URL") (or, optionally, FILE="$(basename "$URL")", wget -cO "$DEST" "$URL" …, mkdir "$DIR", chmod -R +rwx "$DIR", echo "…" -e > "$DESK", and ln -sf "$BIN/idea.sh" /usr/local/bin/idea. And maybe a couple of others; that was a quick scan. … (Cont’d) — G-Man Says 'Reinstate Monica', Dec 19 '17 at 04:21
(Cont’d) … References: Why does my shell script choke on whitespace or other special characters?, Security implications of failing to quote a variable in bash/POSIX shells, ${variable_name} doesn’t mean what you think it does … and Bash quotes unescaped on command substitution. — G-Man Says 'Reinstate Monica', Dec 19 '17 at 04:21

downloading a file behind a link

2 Answers2