How can I best copy large numbers of small files over scp?

Question

I have a directory that's got several gigabytes and several thousand small files. I want to copy it over the network with scp more than once. CPU time on the source and destination machines is cheap, but the network overhead added by copying each file individually is huge. I would tar/gzip it up and ship it over, but the source machine is short on disk.

Is there a way for me to pipe the output of tar -czf <output> <directory> to scp? If not, is there another easy solution? My source machine is ancient (SunOS) so I'd rather not go installing things on it.

Answer 1

You can pipe tar across an ssh session:

$ tar czf - <files> | ssh user@host "cd /wherever && tar xvzf -"

Answer 2

Tar with bzip2 compression should take as much load off the network and on the cpu.

$ tar -C /path/to/src/dir -jcf - ./ | ssh user@server 'tar -C /path/to/dest/dir -jxf -'

Not using -v because screen output might slow down the process. But if you want a verbose output use it on the local side of tar (-jcvf), not on the remote part.

If you repeatedly copy over the same destination path, like updating a backup copy, your best choice is rsync with compression.

$ rsync -az -e ssh /path/to/src/dir/ user@server:/path/to/dest/dir/

Notice that both src and dest paths end with a /. Again, not using -v and -P flags on purpose, add them if you need verbose output.

Answer 3

use rsync, it uses SSH.

Usage:

rsync -aPz /source/path destination.server:remote/path

The rsync switches care about compression and I-Node information. -P displays progress of every file.

You can use scp -C, which enables compression, but if possible, use rsync.

Answer 4

@pdo's answer is good, but one can increase the speed with a buffer and good compression and add a progress bar.

Often the network is the bottleneck and the speed varies over time. Therefore, it helps to buffer the data before sending them over the network. This can be done with with pv.

Additionally, one can usually increase the speed with a proper compression algorithm. Gzip (like used above) is a fast compression algorithm, but in general zstandard (zstd) (and for high compression ratios LZMA/LZMA2 (xz) will compress better and be faster the same time. New xz and zstd have multi core support already built in. To use gzip with multiple cores pigz can be used.

Here is an example to send data with a progress bar, buffering and zstandard compression over a network:

tar cf - . | pv -perabs $(du -sk . | cut -f 1)K | zstd -14 --long=31 -T0 | pv -qCB 512M | ssh user@host "cd /wherever && pv -qCB 512M | zstd -cd -T0 --long=31 | tar xf -"

The first pv is to show the progress (p), estimated time (e), transfer rate (r), average rate (a), total transferred bytes (b). The total size is estimated with du and added to the size option (s). The progress is measured before compression and buffering, therefore it's not very accurate, but still helpful.

zstd is used with the compression setting 14. This number can be reduced or increased depending on the network and CPU speed so zstd is a little bit faster than the the network speed. With four cores on a Haswell 3.2 GHz CPU 14 gives a speed of around 120 MB/s. In the example, the long mode 31 (uses a 2 GB window, needs a lot of RAM, but very good e.g. to compress database dumps) is used. The T0 options sets the amount of threads to the number of cores. One should be aware that together with the long mode these settings use a lot of memory.

A problem with zstd is that most operating systems don't ship with version >= 1.3.4. This version is necessary for proper multi core and long support. If not available, it can be compiled and installed from https://github.com/facebook/zstd with just make -j4 && sudo make install. Instead of zstd, one can also use xz or pigz. xz is slow but compresses very well (good over slow connections), pigz/gzip is fast but compresses not so well. pv is then used again, but for buffering (q for quiet, C for the no splice mode [always needed for buffering] and B to set the buffer size).

In the example a buffer is also used on the receiver side. This is often unnecessary (because decompression and hard disk writing speed is most time higher than the network speed), but does usually not harm, either.

Answer 5

Or you can do it the other way around if you need to. That is pull the tarball over the network rather than push it like has been suggested. This doesn't solve the repeating part of your question and rsync is best for that but there is probably tar switches to help.

So on the local machine:

ssh remote 'tar zcf - /etc/resolv.conf' | tar zxf -

Best to be in the right directory first or you have to use the -C switch on the untaring command at the end.

Just mentioning this in case this is needed. It is for me as in my situation my local server is behind nat, so would take some network futzing about to be able to do it the way that's previously been mentioned.

Answer 6

You can run tar on both ends using ssh. scp is part of the ssh family of goodness, so you probably have it on both ends.

 8:03AM 12 % tar cf - some_directory | ssh dest_host "tar xf -"

There may be a way to work gzip or bzip2 into the pipeline to lessen the network traffic, too.

Answer 7

If you have gzip on both ends: sourcehost$ cd sourcedir && tar cf - . | gzip -c - | ssh user@destinationhost "cd destinationdir && gzip -c -d | tar xf -"

If you don't have gzip on the source machine, make sure you have uncompress on the destination: sourcehost$ cd sourcedir && tar cf - . | compress | ssh user@destinationhost "cd destdir && uncompress | tar xf -"

This would be faster than first zipping it up, then sending, then unzipping, and it requires no extra disk space on either side. I sikpped the compression (z) flag on tar, because you probably dont have it on the ancient side.

Answer 8

Or mount the remote filesystem via sshfs

sshfs user@remotehost:/path/on/remote /path/on/local

Answer 9

While not the most elegant, especially since it's not copying a single zip or tar file and doubly so as it doesn't help to reduce network ovehead, my only choice was to use scp -r:

-r

Recursively copy entire directories.  Note that scp follows symbolic links encountered in the tree traversal.

^{Source: scp(1)}

I was running into issues with running out of disk space with a 30 GB zipped tar file. I thought gunzip could do it inline, i.e., removing the original as it was being unzipped (and I may have missed a Google result) but I couldn’t find anything.

Finally, because I was tired of trying multiple times waiting for a new TAR or ZIP file to be finished tar'ing or zipping I finally just did:

1. From original server/PC/laptop, navigate to the directory where your folder with numerous files/folders are.
2. scp -r source_folder_name yourname@yourservername:destination_folder_name

Then just grab some beer, coffee or popcorn and wait. Good thing is, scp will retry if the network connection "stalls". Just hope it doesn’t go down completely.

Answer 10

This answer is almost 12 years old, however I am using ssh transfer a lot and I am going to share some experiences.

1. It is possible to use scp for transfer whole directory, however it has disadvantages compare to rsync. I was using scp for short time before i replaced scp with rsync. Package rsync should be almost at every Linux computer or server.

2. rsync work for synchronize whole directory with sub-directories. It can be used for synchronize two directories at different places on one computer-server or synchronize two directories, one directory is local and second directory at remote host. Source or destination can be remote or local (local = at server or computer, where you have accounts).

Command Structure is:

rsync -ahv -e ssh --delete --progress --bwlimit=2048k RemoteUser@RemoteHost:/Address_Of_Source_Directory/ /Address_Of_Destination_Directory

Address (full path) of source or target directory begin always with /

RemoteUser@RemoteHost: before path is for remote source-destination. This is not used for local source-destination.

Now lets look to commands:

rsync -ahv -e ssh --delete --progress --bwlimit=2048k Source_Full_Path Target_Full_Path

-e ssh it means that you use ssh protocol for transfering from-to remote host.

--progress showing, how rsync command is running (file by file)

--delete files and directories, that are at destination, but not at source, will be deleted.

--bwlimit=2048k it is limit of transfer speed in kB, to avoid traffic jam of internet network. Value can be changed and this do not need to be in command.

To make sure, that content of source and destination is the same, you can run rsync firs with -ahv (it test files and directories by date of change) and second time with -ahvc, it makes checksum for every file at source and destination (probably sha256sum), and you are sure that content of all files is the same. It can take longer time with plenty of files or many many gigabites of data.

rsync works good for lot of files, however you can compress whole directory with directory structure. Tar is not good if names of files are not only alphanumeric. Better is 7z (p7zip). Go to directory with data and then start 7zip. Best compression results are probably:

7z a -t7z -m0=lzma2 -mqs -mx=9 -mfb=273 -md=64m -ms=on -mmt=2 Name_of_archive.7z

-mqs it means that files in row are sorted by type (txt jpg pdf etc), not by directory structure and names. It can get better results.

-md=64m Something like size of blocks, dictationary. 64m is enough (64MB). You can add more, but it takes at least 11-12 times more RAM than size of blocks.

-mmt=2 numer of CPU treads. Use -mmt=off for one tread (if you have lot of small files) or -mmt=2. If you use more that 2 treads, it will takes more RAM. 7z takes ram something like this (numer of pairs of CPU treads * block-dictatonary size)*(around 11-12 times) or if you have lot of files (milions), it takes few more GiB of RAM.

Answer 11

I have destination that needs PEM to connect and it also has different SSH port. This worked for me like charm:

For e.g. Copy folder (and subfolders) from sourceserver to targetserver, run this on sourceserver

$ rsync -azu -e 'ssh -i ./pem_for_target_server.pem -p <port_number_for_target_server>' /folder/path/on/source username_on_target@target.server.name:/path/on/target/where/you/want/to/copy/folder/on/source/

(In case you get permissions are too open for pem file, run this: chmod 400 ./pem_for_target_server.pem)

Answer 12

These is what I got after having read all above answers:

tarscp_from_remote() {
    if [ "$#" -ne 3 ]; then echo "Usage: tarscp_from_remote {remote_host} {remote_dir_path} {local_dir_path}" >&2; return; fi
    ssh $1 "tar -C $2 -cf - ./" | tar -C $3 -xf -
}
tarscp_to_remote() {
    if [ "$#" -ne 3 ]; then echo "Usage: tarscp_to_remote {local_dir_path} {remote_host} {remote_dir_path}" >&2; return; fi
    tar -C $1 -cf - ./ | ssh $2 "tar -C $3 -xf -"
}

And the version with compression (xz):

tarscp_from_remote() {
    if [ "$#" -ne 3 ]; then echo "Usage: tarscp_from_remote {remote_host} {remote_dir_path} {local_dir_path}" >&2; return; fi
    ssh $1 "tar -C $2 -Jcf - ./" | tar -C $3 -Jxf -
}
tarscp_to_remote() {
    if [ "$#" -ne 3 ]; then echo "Usage: tarscp_to_remote {local_dir_path} {remote_host} {remote_dir_path}" >&2; return; fi
    tar -C $1 -Jcf - ./ | ssh $2 "tar -C $3 -Jxf -"
}

Answer 13

DIRECTORY you wrote - this is overlooked by many good answers. YES, you can copy a distinct directory including subsequent content with tar (NOT tampering with owners and rights in the directory):

• become root if necessary
• FROM: copy dir "gereric_horse" without path ("/mnt/piglet/")
• TO: into directory "/mnt/moon/"
• no need to change into a particular dir first

Example:

tar --one-file-system --atime-preserve --preserve-permissions --exclude-caches --exclude='.Trash*' -cf - -C /mnt/piglet/ gereric_horse | (cd /mnt/moon/ && tar --atime-preserve --preserve-permissions -xvf - ) 

The ( ) means: Subshell. Better results.

This way you copy a directory with tar. Works fine ;-)
BTW: rsync should be considered as well

---

I was hinted the "remote" part is missing; admitted ;-)

Here it comes as an example you may adopt to fit your env:
tar from local to remote

[*part before pipe*]    |  ssh  -l user4ssh  2a03:2260:6873:d1ff:4bf9:c9c4:d80d:5bde  'cd /home/targetman/Downloads/ && tar --atime-preserve --preserve-permissions  -xvf - '

Please take care to have matching access rights
No need for additional subshell "()" here,
neither "[" around IP address;
necessary to have single quotation marks around the "cd..." term.
SSH format needs separated target-user option behind the "-l"
The ip address is anonymized but the entire format is taken from a real working example.

Here comes the other way round:
local invoked tar from remote to local (w/o separating dir stuff)

sudo ssh remoteuser@2a03:2345:6873:d1ff:4bf9:c9c4:d80d:5bde  sudo -S tar --same-owner --one-file-system  --atime-preserve --preserve-permissions --exclude-caches --exclude='.Trash*'  -cf - /sourcedir/ | (cd /mnt/targetdir/ && sudo -S tar --atime-preserve --preserve-permissions --same-owner  -xvf - )

I did not invent the wheel. I borrowed it here:
https://docstore.mik.ua/orelly/unix3/upt/ch10_13.htm

PS: If this may be helpful please consider to upvote my contrib.