Downloading files over ssh based on a list of filenames

Question

So here is what I have done already:

I have a file called abc.txt which contains list of files. I am using abc.txt to move those files to a folder, tar that folder and finally I download the tar to my local PC from my server, which is running GNU/Linux.

Here are the steps in list form:

abc.txt
abc.txt (listed files) -> folder
Folder -> folder.tar
folder.tar -> local PC.

If abc.txt contains 2 files, for example:

example1.css
example2.css

I need to download those files from abc.txt separately and directly to the local PC.

Since ftp or sftp need the file name to download it, how can I read that from abc.txt?

you download the file read it locally and redownload the right file ? — Kiwy, Dec 30 '13 at 11:37
or you use ssh to execute the script on your remote machine. — Kiwy, Dec 30 '13 at 11:37

janos · Answer 1 · 2014-01-01T21:35:00.160

If the file abc.txt contains the list of filenames relative from /path/to/base:

ssh user@server tar c -T /path/to/abc.txt -C /path/to/base | (cd /tmp; tar xv)

This creates a tarball on-the-fly, without actually saving it anywhere, pipe it to the local shell, extract, effectively copying the listed files.

EXTRA TIPS

If the file abc.txt contains the list of absolute paths:

ssh user@server tar c -T /path/to/abc.txt | (cd /tmp; tar xv)

If the file abc.txt is on your local system, not the remote:

ssh user@server tar c -T- < /path/to/abc.txt | (cd /tmp; tar xv)

To use gzip compression (using default level 6):

ssh -C user@server tar c -T /path/to/abc.txt | (cd /tmp; tar xv)

To use gzip compression level 9:

ssh user@server 'tar c -T /path/to/abc.txt | gzip -9' | (cd /tmp; tar zxv)

score 1 · Answer 2 · answered Dec 30 '13 at 11:49

1

Here is a simple script witch works with files containing spaces in the filename :

SAVEIFS=$IFS; IFS=$(echo -en "\n\b") ; for elt in `cat abc.txt` ; do scp $elt <YOUR_SFTP_CONNECTION> ; done ; IFS=$SAVEIFS

answered Dec 30 '13 at 11:49

Slyx

3,885

Why treat backspace as a separator? (Yes, I know you read it somewhere, but I've never traced it back to its origin. It makes no sense.) You forgot to turn off globbing. Using read is more robust, see How to loop over the lines of a file?. – Gilles 'SO- stop being evil' Dec 30 '13 at 22:57

score 0 · Answer 3 · answered Dec 30 '13 at 11:45

I think the hub of your problem is how to extract the correct files from your list for your subsequent two logic paths.

egrep 'example1.css|example2.css' abc.txt

will give you all lines that match the exceptions, and

egrep -v 'example1.css|example2.css' abc.txt

will give you all lines that don't match the exceptions

Downloading files over ssh based on a list of filenames

3 Answers3