Download url from CSV file and rename the file

Question

I have a csv file with 3 columns as follow, I need to create a batch file to download the images in the URLs from a FTP server to a folder and rename them to a new name from column A.

New_Name,URL,Status   
MyName1.jpg,ftp://images:Img12345@FTP-Location.com/images/image01.jpg,File Downloaded   
MyName2.jpg,ftp://images:Img12345@FTP-Location.com/images/image02.jpg,File Downloaded 
MyName3.jpg,ftp://images:Img12345@FTP-Location.com/images/image03.jpg,File Not Found

It would really be fantastic, if it could write the status of the download back to the csv file! If the file was downloaded successfully it would write "File Downloaded" it in column C (Status), otherwise it would write "File Not Found".

Is this possible?

For starter, I utilized the following script, called test.sh, issued the command sh test.sh at the command prompt, just to download the URLs, without any luck!

#!/bin/sh   
for link in `cat test.csv | cut -d, -f2`     
do   
     wget $link -O /mnt/nas_1tb/a-test/   
done

I found the above code from here.

score 1 · Answer 1 · edited Apr 13 '17 at 12:36

You appear to want to download the file into a particular directory. You can't do it by passing the name of the directory to wget. Either pass a full file name, or let wget pick the name from the link. If wget picks the name, it saves the file in the current directory, so change to the desired directory first.

Parsing the list of URLs with a command substitution is fragile. It might fail if URLs contain ? and will fail if they contain spaces. You can use read to process the list line by line.

csv_file="$PWD/test.csv"
cd /mnt/nas_1tb/a-test/
while IFS=, read -r column1 url trail; do
  wget "$url"
done <"$csv_file"

Since you want to pick the file names, you should pass the -O option. But pass the whole file name.

To update the download status in the third column, create a new file with the updated data.

while IFS=, read -r new_name url status; do
  wget -q -O "/mnt/nas_1tb/a-test/$new_name" "$url"
  case $? in
    0) status='File fownloaded';;
    8) status='File not found or server error';;
    *) status='Download failed';;
  esac
  printf '%s,%s,%s\n' "$new_name" "$url" "$status"
done <test.csv >test-results.csv

Note that wget doesn't make it easy to distinguish “file not found” (HTTP 404) from other errors (e.g. HTTP 403 not authorized, HTTP 500 server internal error, etc.). But you do at least get different errors for errors when the server couldn't return a response.

Download url from CSV file and rename the file

1 Answers1