56

I am running Mac OS 10.11.6 El Capitan. There is a link I would like to download programmatically:

https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.16-osx10.11-x86_64.dmg

If I paste this URL into any browser (e.g. Safari) the download works perfectly.

However, if I try to download the same URL from the command line using curl, it doesn't work—the result is an empty file:

$ ls -lA
$ curl -O https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.16-osx10.11-x86_64.dmg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
$ ls -lA
total 0
-rw-r--r--  1 myname  staff  0 Nov  7 14:07 mysql-5.7.16-osx10.11-x86_64.dmg
$

Of course I can get the file through the browser, but I would like to understand why the curl command above doesn't work.

Why can't curl download this file correctly, when it is evidently present on the website and can be correctly accessed and downloaded through a graphical web browser?

Braiam
  • 35,991
mattobob
  • 693
  • 7
    I'd like to point out that even though techraf's answer on redirects is completely correct, other factors such as headers may cause a server to reject the curl client's request to download a file.

    For example, if the server has backend DDoS protection, such protection software commonly checks for viable browser headers, such as having a proper User-Agent. Additionally, some browser downloads may succeed due to session cookies (i.e if you're logged in) being only present on said browser.

     – Joseph Nov 08 '16 at 10:15
  • 8
    To troubleshoot a curl command, you can use curl -v for "verbose". It will print to standard error various info about the connection, request, and response. In this case, you would see that the response includes HTTP 302 Found (a redirect code) and a Location header with the URL to go to. Then you could man curl to find out how to tell it to follow redirects. – Nathan Long Nov 08 '16 at 17:01
  • I had the same problem and adding authorization header worked for me. curl --header "Authorization: Basic 12345" -o indexexport.zip http://localhost:8080/indexexport – Andrei Jul 20 '22 at 14:15

3 Answers3

97

There is a redirect on the webserver-side to the following URL: http://cdn.mysql.com//Downloads/MySQL-5.7/mysql-5.7.16-osx10.11-x86_64.dmg. Because it's a CDN, the exact behaviour (whether you get redirected or not) might depend on your location.

curl does not follow redirects by default. To tell it to do so, add the -L argument:

curl -L -O https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.16-osx10.11-x86_64.dmg
techraf
  • 5,941
18

If the browser is able to download the file, you can inspect what the browser is doing. On google chrome you can use the following to see what’s happening.

1) [View > Developer > Developer tools > Network Tab > Headers tab]

2) Click on the download link.

3) The file link will appear on the developer tools tab.

4) Right click on the file and select Copy > Copy as cURL.

Now you have a curl link that will work. It will probably have excess parameters you can trim-away.

More details: https://lornajane.net/posts/2013/chrome-feature-copy-as-curl

cage
  • 189
  • 2
    YES! This is exactly what I needed. Maybe it's obvious, but I took the cURL command and added--output download.csv to the end of it, else it streams the file to stdout I think. – Nick Crews Feb 17 '22 at 00:43
  • won't work in many cases, curl from chrome doesn't work, but the browser still render the jpeg file – TomSawyer Dec 14 '22 at 16:23
1

I am going to convert one of the comments on this post to an answer.

There are many HTTP/HTTPS links that require certain headers in order to work. So this will result in a working response from a web browser but not a working response in a backend web request like curl.

I just ran into a site that required all the following headers. Failure to specify them resulted in a timeout.

  httpget.setHeader("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36");
  httpget.setHeader("Upgrade-Insecure-Requests", "1");
  httpget.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
  httpget.setHeader("Accept-Encoding", "gzip, deflate, br");
  httpget.setHeader("Accept-Language", "en-US,en;q=0.9");
  httpget.setHeader("Connection", "keep-alive");
  httpget.setHeader("Host", "www.thehost.com");
Nicholas DiPiazza
  • 438